[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:26 dist-test-slave-fcdn krb5kdc[28352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:26 dist-test-slave-fcdn krb5kdc[28352](info): set up 2 sockets
Sep 25 19:09:26 dist-test-slave-fcdn krb5kdc[28352](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 19:09:27.620522 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:27 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:27.752815 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43311
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:27.836205 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:27.836410 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:27.836432 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:27.837512 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:27.837543 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:27.837555 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:27.837568 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:27.837580 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:27.838938 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:27.839124 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:27.839294 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:27.844183 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:27.844192 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:27.844192 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:27.844465 28365 server_base.cc:1034] running on GCE node
I20260925 19:09:27.844702 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:27.844991 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:27.846159 28365 hybrid_clock.cc:648] HybridClock initialized: now 1790363367846152 us; error 54 us; skew 500 ppm
Sep 25 19:09:27 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:27.848730 28365 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:27.849159 28365 webserver.cc:460] Webserver started at http://127.27.174.126:33515/ using document root <none> and password file <none>
I20260925 19:09:27.849382 28365 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:27.849457 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:27.849607 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:27.850621 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "e2adda019bc34ddebe56c696bbd1d1c6"
format_stamp: "Formatted at 2026-09-25 19:09:27 on dist-test-slave-fcdn"
server_key: "78290e4a1ee79ec14bc7726102064848"
server_key_iv: "7df8434e0f5bb4503f9525cf9590e5ae"
server_key_version: "encryptionkey@0"
I20260925 19:09:27.850991 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "e2adda019bc34ddebe56c696bbd1d1c6"
format_stamp: "Formatted at 2026-09-25 19:09:27 on dist-test-slave-fcdn"
server_key: "78290e4a1ee79ec14bc7726102064848"
server_key_iv: "7df8434e0f5bb4503f9525cf9590e5ae"
server_key_version: "encryptionkey@0"
I20260925 19:09:27.852393 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:27.853137 28380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:27.853303 28365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:27.853408 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "e2adda019bc34ddebe56c696bbd1d1c6"
format_stamp: "Formatted at 2026-09-25 19:09:27 on dist-test-slave-fcdn"
server_key: "78290e4a1ee79ec14bc7726102064848"
server_key_iv: "7df8434e0f5bb4503f9525cf9590e5ae"
server_key_version: "encryptionkey@0"
I20260925 19:09:27.853480 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:27.860972 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:27.863209 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:27.863366 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:27.867174 28365 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43311
I20260925 19:09:27.867224 28432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43311 every 8 connection(s)
I20260925 19:09:27.867583 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:27.868489 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:27.870728 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6: Bootstrap starting.
I20260925 19:09:27.871325 28433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:27.871666 28433 log.cc:824] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:27.872350 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6: No bootstrap required, opened a new log
I20260925 19:09:27.873981 28433 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } }
I20260925 19:09:27.874115 28433 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:27.874137 28433 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2adda019bc34ddebe56c696bbd1d1c6, State: Initialized, Role: FOLLOWER
I20260925 19:09:27.874305 28433 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } }
I20260925 19:09:27.874392 28433 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:27.874444 28433 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:27.874502 28433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:27.875062 28433 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } }
I20260925 19:09:27.875172 28433 leader_election.cc:302] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2adda019bc34ddebe56c696bbd1d1c6; no voters:
I20260925 19:09:27.875357 28433 leader_election.cc:288] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:27.875415 28436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:27.875569 28436 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [term 1 LEADER]: Becoming Leader. State: Replica: e2adda019bc34ddebe56c696bbd1d1c6, State: Running, Role: LEADER
I20260925 19:09:27.875633 28433 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:27.875672 28436 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } }
I20260925 19:09:27.876060 28438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2adda019bc34ddebe56c696bbd1d1c6. Latest consensus state: current_term: 1 leader_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } } }
I20260925 19:09:27.876145 28438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:27.876181 28437 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2adda019bc34ddebe56c696bbd1d1c6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43311 } } }
I20260925 19:09:27.876248 28437 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:27.876386 28444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:27.876933 28444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:27.877751 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28365
I20260925 19:09:27.877853 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:27.878624 28345 external_mini_cluster.cc:1428] Setting key 5203246034cdb4eb61ed584b282c6262
I20260925 19:09:27.880672 28444 catalog_manager.cc:1347] Generated new cluster ID: d480977a7a7240269e524e4e9322d39c
I20260925 19:09:27.880726 28444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:27 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:27.895987 28444 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:27.896670 28444 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:27.897030 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:27.879564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.27.174.126:43311)
0925 19:09:27.879973 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:27.879981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:27.880063 (+ 82us) server_negotiation.cc:407] Connection header received
0925 19:09:27.882415 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:27.882422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:27.882479 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:27.882620 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:27.884604 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:27.886444 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:27.887950 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:27.888558 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:27.888673 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:27.892871 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:27.892875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:27.892876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:27.892887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:27.894877 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:27.895393 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:27.895396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:27.895396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:27.895490 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:27.895834 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:27.895835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:27.895836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:27.896157 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:27.896352 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:27.896622 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 19:09:27.896769 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":89,"threads_started":1}
I20260925 19:09:27.916004 28444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2adda019bc34ddebe56c696bbd1d1c6: Generated new TSK 0
I20260925 19:09:27.916258 28444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:27.950392 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:28.031221 28459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:28.031421 28459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:28.031445 28459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:28.032653 28459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:28.032696 28459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:28.032725 28459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:28.034237 28459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:28.034466 28459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:28.034716 28459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:28.039597 28464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:28.039693 28465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:28.039696 28467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:28.040141 28459 server_base.cc:1034] running on GCE node
I20260925 19:09:28.040344 28459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:28.040620 28459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:28.041781 28459 hybrid_clock.cc:648] HybridClock initialized: now 1790363368041757 us; error 44 us; skew 500 ppm
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:28.044202 28459 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:28.044658 28459 webserver.cc:460] Webserver started at http://127.27.174.65:44519/ using document root <none> and password file <none>
I20260925 19:09:28.044899 28459 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:28.044968 28459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:28.045109 28459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:28.046126 28459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "feac2946b486450eb966051e862e8260"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "a873e6eee9608c44e1a85cf7eea15fe7"
server_key_iv: "ecaf27aa672f6f615c8ea301828a247d"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.046482 28459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "feac2946b486450eb966051e862e8260"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "a873e6eee9608c44e1a85cf7eea15fe7"
server_key_iv: "ecaf27aa672f6f615c8ea301828a247d"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.047849 28459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:09:28.048583 28474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.048743 28459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:28.048859 28459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "feac2946b486450eb966051e862e8260"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "a873e6eee9608c44e1a85cf7eea15fe7"
server_key_iv: "ecaf27aa672f6f615c8ea301828a247d"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.048933 28459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:28.066468 28459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:28.068414 28459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:28.068584 28459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:28.068821 28459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:28.069135 28459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:28.069190 28459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.069232 28459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:28.069257 28459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.075632 28459 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39077
I20260925 19:09:28.075659 28587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39077 every 8 connection(s)
I20260925 19:09:28.076074 28459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:28.085299 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28459
I20260925 19:09:28.085405 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:28.085552 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.076773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43259 (local address 127.27.174.126:43311)
0925 19:09:28.076821 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:28.076831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:28.077061 (+ 230us) server_negotiation.cc:407] Connection header received
0925 19:09:28.077662 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:28.077663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:28.077673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:28.077925 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:28.078835 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.079331 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.080358 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.080553 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.080610 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.082631 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:28.082632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:28.082632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:28.082638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:28.084048 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.084406 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.084407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.084407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.084458 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.084693 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.084694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.084694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.084837 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:28.084900 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:28.085346 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 19:09:28.085455 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:28.085562 28345 external_mini_cluster.cc:1428] Setting key 8259ccc4c34aa66ecb8276ddc48b75cd
I20260925 19:09:28.085685 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.076768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43311 (local address 127.27.174.65:43259)
0925 19:09:28.076934 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:28.076973 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:28.077562 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:28.077989 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:28.077992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:28.078536 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:28.078721 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.078725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.079414 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.079414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.080276 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.080278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.080626 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.080626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.080720 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.081184 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:28.081192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:28.082535 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:28.084123 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.084128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.084130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.084340 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.084523 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.084523 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.084524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.084614 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.084941 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:28.084946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:28.085095 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 19:09:28.085260 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 19:09:28.085456 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260925 19:09:28.086239 28588 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43311
I20260925 19:09:28.086339 28588 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:28.086520 28588 heartbeater.cc:511] Master 127.27.174.126:43311 requested a full tablet report, sending...
I20260925 19:09:28.087087 28397 ts_manager.cc:194] Registered new tserver with Master: feac2946b486450eb966051e862e8260 (127.27.174.65:39077)
I20260925 19:09:28.087960 28397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:43259
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:28.110870 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:28.191128 28593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:28.191329 28593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:28.191354 28593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:28.192453 28593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:28.192489 28593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:28.192520 28593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:28.193882 28593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:28.194057 28593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:28.194216 28593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:28.199188 28598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:28.199231 28601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:28.199185 28599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:28.199431 28593 server_base.cc:1034] running on GCE node
I20260925 19:09:28.199774 28593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:28.200011 28593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:28.201164 28593 hybrid_clock.cc:648] HybridClock initialized: now 1790363368201143 us; error 39 us; skew 500 ppm
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:28.203795 28593 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:28.204221 28593 webserver.cc:460] Webserver started at http://127.27.174.66:42575/ using document root <none> and password file <none>
I20260925 19:09:28.204459 28593 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:28.204504 28593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:28.204624 28593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:28.205652 28593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "fd186f81e9e74214aababce764bc6ef7"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "3c5a7b1f2c318fe1a810535e917e1534"
server_key_iv: "5412de8ebf36a3e3d76b0d45eb10cdb2"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.206003 28593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "fd186f81e9e74214aababce764bc6ef7"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "3c5a7b1f2c318fe1a810535e917e1534"
server_key_iv: "5412de8ebf36a3e3d76b0d45eb10cdb2"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.207278 28593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:28.208062 28608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.208259 28593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:09:28.208341 28593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "fd186f81e9e74214aababce764bc6ef7"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "3c5a7b1f2c318fe1a810535e917e1534"
server_key_iv: "5412de8ebf36a3e3d76b0d45eb10cdb2"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.208393 28593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:28.233376 28593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:28.235343 28593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:28.235486 28593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:28.235744 28593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:28.236055 28593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:28.236110 28593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.236152 28593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:28.236179 28593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.241756 28593 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:35147
I20260925 19:09:28.241832 28721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:35147 every 8 connection(s)
I20260925 19:09:28.242147 28593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:28.245625 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28593
I20260925 19:09:28.245736 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:28.245914 28345 external_mini_cluster.cc:1428] Setting key 16705135061ba5cb823a7974bb543f1e
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:28.251477 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.242883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:50763 (local address 127.27.174.126:43311)
0925 19:09:28.242927 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:28.242929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:28.243068 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:09:28.243566 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:28.243567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:28.243579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:28.243680 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:28.244576 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.245090 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.246137 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.246370 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.246439 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.248625 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:28.248626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:28.248627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:28.248634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:28.250041 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.250465 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.250466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.250467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.250528 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.250785 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.250786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.250787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.250899 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:28.250973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:28.251244 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 19:09:28.251332 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:28.251708 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.242792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43311 (local address 127.27.174.66:50763)
0925 19:09:28.242953 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:28.242993 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:28.243456 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:28.243745 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:28.243749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:28.244298 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:28.244486 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.244491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.245163 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.245164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.246071 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.246074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.246507 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.246507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.246611 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.247109 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:28.247117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:28.248519 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:28.250153 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.250159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.250161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.250386 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.250595 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.250596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.250597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.250708 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.250977 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:28.250983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:28.251116 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 19:09:28.251224 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:09:28.251380 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260925 19:09:28.252185 28722 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43311
I20260925 19:09:28.252295 28722 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:28.252465 28722 heartbeater.cc:511] Master 127.27.174.126:43311 requested a full tablet report, sending...
I20260925 19:09:28.252889 28397 ts_manager.cc:194] Registered new tserver with Master: fd186f81e9e74214aababce764bc6ef7 (127.27.174.66:35147)
I20260925 19:09:28.253439 28397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:50763
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:28.271108 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:28.350858 28727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:28.351053 28727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:28.351078 28727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:28.352326 28727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:28.352367 28727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:28.352394 28727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:28.353844 28727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:28.354094 28727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:28.354321 28727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:28.359421 28735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:28.359406 28732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:28.359515 28727 server_base.cc:1034] running on GCE node
W20260925 19:09:28.359407 28733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:28.359917 28727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:28.360162 28727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:28.361335 28727 hybrid_clock.cc:648] HybridClock initialized: now 1790363368361328 us; error 42 us; skew 500 ppm
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:28.363886 28727 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:28.364374 28727 webserver.cc:460] Webserver started at http://127.27.174.67:44801/ using document root <none> and password file <none>
I20260925 19:09:28.364615 28727 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:28.364681 28727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:28.364817 28727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:28.365867 28727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "b02e6ca2a112487eb47f3423d45a87bb"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "86945aa7d6d9820e4b6435d65c25ca21"
server_key_iv: "f18773c385fb470bed97be97165b6173"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.366228 28727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "b02e6ca2a112487eb47f3423d45a87bb"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "86945aa7d6d9820e4b6435d65c25ca21"
server_key_iv: "f18773c385fb470bed97be97165b6173"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.367544 28727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:28.368311 28742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.368481 28727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:28.368578 28727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "b02e6ca2a112487eb47f3423d45a87bb"
format_stamp: "Formatted at 2026-09-25 19:09:28 on dist-test-slave-fcdn"
server_key: "86945aa7d6d9820e4b6435d65c25ca21"
server_key_iv: "f18773c385fb470bed97be97165b6173"
server_key_version: "encryptionkey@0"
I20260925 19:09:28.368660 28727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:28.380752 28727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:28.383102 28727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:28.383270 28727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:28.383488 28727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:28.383850 28727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:28.383903 28727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.383945 28727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:28.383970 28727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:28.390075 28727 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:34013
I20260925 19:09:28.390111 28855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:34013 every 8 connection(s)
I20260925 19:09:28.390444 28727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:28.395799 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28727
I20260925 19:09:28.395895 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:28.396095 28345 external_mini_cluster.cc:1428] Setting key acbe708dfcf3a824614e1ffc760fe00b
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:28.399993 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.391261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:58391 (local address 127.27.174.126:43311)
0925 19:09:28.391340 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:28.391342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:28.391488 (+ 146us) server_negotiation.cc:407] Connection header received
0925 19:09:28.392049 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:28.392050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:28.392060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:28.392111 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:28.393022 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.393513 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.394532 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.394728 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.394797 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.397118 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:28.397120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:28.397120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:28.397128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:28.398518 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.398945 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.398947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.398947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.399009 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.399270 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.399271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.399271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.399390 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:28.399474 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:28.399821 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 19:09:28.399898 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:09:28.400192 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.391208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43311 (local address 127.27.174.67:58391)
0925 19:09:28.391362 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:28.391397 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:28.391961 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:28.392166 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:28.392169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:28.392756 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:28.392947 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.392952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.393611 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.393612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.394445 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:28.394447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.394818 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.394818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:28.394928 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.395389 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:28.395397 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:28.397022 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:28.398619 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.398626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.398628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.398853 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.399077 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:28.399078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:28.399079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:28.399185 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:28.399465 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:28.399470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:28.399665 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 19:09:28.399800 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 19:09:28.399964 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260925 19:09:28.400610 28856 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43311
I20260925 19:09:28.400689 28856 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:28.400835 28856 heartbeater.cc:511] Master 127.27.174.126:43311 requested a full tablet report, sending...
I20260925 19:09:28.401149 28397 ts_manager.cc:194] Registered new tserver with Master: b02e6ca2a112487eb47f3423d45a87bb (127.27.174.67:34013)
I20260925 19:09:28.401475 28397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:58391
I20260925 19:09:28.408244 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:28.430812 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.423070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.27.174.65:39077)
0925 19:09:28.423250 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:28.423255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:28.423265 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:28.423299 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:28.423301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:28.423314 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:28.423396 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:28.424038 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.424745 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.425475 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.425803 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.425880 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.428186 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:28.428193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:28.428194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:28.428201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:28.429662 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.430019 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.430022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.430022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.430080 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.430274 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.430275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.430275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.430407 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:28.430562 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:28.430615 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:09:28.430692 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
W20260925 19:09:28.431242 28502 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59042
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:28.452976 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:28.445850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.27.174.65:39077)
0925 19:09:28.445913 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:28.445915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:28.445990 (+ 75us) server_negotiation.cc:407] Connection header received
0925 19:09:28.446094 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:28.446095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:28.446107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:28.446180 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:28.446722 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.447214 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.448012 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:28.448216 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:28.448288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:28.450335 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:28.450336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:28.450337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:28.450342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:28.451803 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.452191 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.452193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.452193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.452263 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:28.452461 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:28.452462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:28.452462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:28.452561 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:28.452691 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:28.452794 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:09:28.452871 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:28.453526 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28459
I20260925 19:09:28.458647 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28593
I20260925 19:09:28.462965 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28727
I20260925 19:09:28.467018 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28365
2026-09-25T19:09:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1918 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28870](info): set up 2 sockets
Sep 25 19:09:28 dist-test-slave-fcdn krb5kdc[28870](info): commencing operation
krb5kdc: starting...
W20260925 19:09:29.511111 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:29 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:29.624607 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34513
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:29.706773 28883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:29.707001 28883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:29.707039 28883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:29.708268 28883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:29.708323 28883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:29.708350 28883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:29.708364 28883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:29.708376 28883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:29.709753 28883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:29.709931 28883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:29.710088 28883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:29.714673 28889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:29.714727 28891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:29.714843 28888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:29.714836 28883 server_base.cc:1034] running on GCE node
I20260925 19:09:29.715150 28883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:29.715420 28883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:29.716564 28883 hybrid_clock.cc:648] HybridClock initialized: now 1790363369716555 us; error 29 us; skew 500 ppm
Sep 25 19:09:29 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:29.718981 28883 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:29.719327 28883 webserver.cc:460] Webserver started at http://127.27.174.126:45907/ using document root <none> and password file <none>
I20260925 19:09:29.719496 28883 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:29.719533 28883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:29.719679 28883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:29.720655 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "a30f2c6f389a496f92656cd66e96e9a8"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "2a96345021770ff4b56254aa64f06d2b"
server_key_iv: "db99d54d693c8da8733893dcdfe11058"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.720949 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "a30f2c6f389a496f92656cd66e96e9a8"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "2a96345021770ff4b56254aa64f06d2b"
server_key_iv: "db99d54d693c8da8733893dcdfe11058"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.722126 28883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:29.722739 28898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:29.722937 28883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:29.723024 28883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "a30f2c6f389a496f92656cd66e96e9a8"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "2a96345021770ff4b56254aa64f06d2b"
server_key_iv: "db99d54d693c8da8733893dcdfe11058"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.723093 28883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:29.738492 28883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:29.740707 28883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:29.740857 28883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:29.745035 28883 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34513
I20260925 19:09:29.745095 28950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34513 every 8 connection(s)
I20260925 19:09:29.745414 28883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:29.746266 28951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:29.748543 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: Bootstrap starting.
I20260925 19:09:29.748916 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28883
I20260925 19:09:29.749010 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:29.749157 28345 external_mini_cluster.cc:1428] Setting key 00bc1e7a0b5d25de9f487e804eda4701
I20260925 19:09:29.749262 28951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:29.749583 28951 log.cc:824] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:29.750309 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: No bootstrap required, opened a new log
I20260925 19:09:29.752141 28951 raft_consensus.cc:348] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } }
I20260925 19:09:29.752301 28951 raft_consensus.cc:374] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:29.752347 28951 raft_consensus.cc:729] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30f2c6f389a496f92656cd66e96e9a8, State: Initialized, Role: FOLLOWER
I20260925 19:09:29.752489 28951 consensus_queue.cc:260] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } }
I20260925 19:09:29.752568 28951 raft_consensus.cc:388] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:29.752609 28951 raft_consensus.cc:482] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:29.752655 28951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:29.753153 28951 raft_consensus.cc:504] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } }
I20260925 19:09:29.753276 28951 leader_election.cc:302] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a30f2c6f389a496f92656cd66e96e9a8; no voters:
I20260925 19:09:29.753465 28951 leader_election.cc:288] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:29.753499 28956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:29.753614 28956 raft_consensus.cc:686] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [term 1 LEADER]: Becoming Leader. State: Replica: a30f2c6f389a496f92656cd66e96e9a8, State: Running, Role: LEADER
I20260925 19:09:29.753706 28951 sys_catalog.cc:564] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:29.753700 28956 consensus_queue.cc:237] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } }
I20260925 19:09:29.754220 28957 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a30f2c6f389a496f92656cd66e96e9a8. Latest consensus state: current_term: 1 leader_uuid: "a30f2c6f389a496f92656cd66e96e9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } } }
I20260925 19:09:29.754352 28957 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:29.754215 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a30f2c6f389a496f92656cd66e96e9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30f2c6f389a496f92656cd66e96e9a8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34513 } } }
I20260925 19:09:29.754678 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:29.755481 28972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:29.755534 28972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:29.755584 28965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:29.755775 28965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:09:29 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:29.757138 28965 catalog_manager.cc:1347] Generated new cluster ID: 7191a3355de7483cb5dd61a621b67597
I20260925 19:09:29.757193 28965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:29.759805 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:29.749698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.27.174.126:34513)
0925 19:09:29.749889 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:29.749896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:29.749920 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:09:29.750321 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:29.750327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:29.750407 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:29.750564 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:29.751149 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.752066 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.753035 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.753546 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.753648 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:29.756764 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:29.756768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:29.756768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:29.756779 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:29.758306 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:29.758667 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:29.758669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:29.758670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:29.758766 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:29.758978 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:29.758978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:29.758979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:29.759207 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:29.759347 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:29.759457 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:09:29.759584 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260925 19:09:29.788924 28965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:29.789731 28965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:29.801179 28965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a30f2c6f389a496f92656cd66e96e9a8: Generated new TSK 0
I20260925 19:09:29.801406 28965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:29.828467 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:29.907718 28977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:29.907907 28977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:29.907929 28977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:29.909060 28977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:29.909111 28977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:29.909137 28977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:29.910611 28977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:29.910831 28977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:29.911065 28977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:29.916096 28983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:29.916106 28982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:29.916275 28977 server_base.cc:1034] running on GCE node
W20260925 19:09:29.916106 28985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:29.916689 28977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:29.916900 28977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:29.918057 28977 hybrid_clock.cc:648] HybridClock initialized: now 1790363369918039 us; error 32 us; skew 500 ppm
Sep 25 19:09:29 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:29.920605 28977 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:29.921044 28977 webserver.cc:460] Webserver started at http://127.27.174.65:38475/ using document root <none> and password file <none>
I20260925 19:09:29.921279 28977 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:29.921325 28977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:29.921450 28977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:29.922493 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "5a4d3e1629d34213b0ea9bec3e770f01"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "052df725b35a3a29499e3a89c1cd27ec"
server_key_iv: "aadc4310d7cfd86b634757933b472c0f"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.922833 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4d3e1629d34213b0ea9bec3e770f01"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "052df725b35a3a29499e3a89c1cd27ec"
server_key_iv: "aadc4310d7cfd86b634757933b472c0f"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.924201 28977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:29.924868 28992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:29.925074 28977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:29.925163 28977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "5a4d3e1629d34213b0ea9bec3e770f01"
format_stamp: "Formatted at 2026-09-25 19:09:29 on dist-test-slave-fcdn"
server_key: "052df725b35a3a29499e3a89c1cd27ec"
server_key_iv: "aadc4310d7cfd86b634757933b472c0f"
server_key_version: "encryptionkey@0"
I20260925 19:09:29.925236 28977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:29.933710 28977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:29.935523 28977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:29.935731 28977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:29.935959 28977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:29.936252 28977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:29.936285 28977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:29.936303 28977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:29.936349 28977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:29.942028 28977 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40475
I20260925 19:09:29.942085 29105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40475 every 8 connection(s)
I20260925 19:09:29.942381 28977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:29.942718 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 28977
I20260925 19:09:29.942812 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:29.943001 28345 external_mini_cluster.cc:1428] Setting key 2f07dd0f9970100363b410a3ebe70dc6
Sep 25 19:09:29 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:09:29.952919 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:29.943155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39555 (local address 127.27.174.126:34513)
0925 19:09:29.943209 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:29.943212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:29.943404 (+ 192us) server_negotiation.cc:407] Connection header received
0925 19:09:29.944690 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:29.944691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:29.944700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:29.944771 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:29.945514 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.946284 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.947310 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.947527 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.947630 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:29.949956 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:29.949958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:29.949958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:29.949965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:29.951459 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:29.951935 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:29.951936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:29.951936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:29.951998 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:29.952330 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:29.952331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:29.952331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:29.952439 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:29.952493 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:29.952784 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 19:09:29.952854 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:09:29.953012 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:29.943136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34513 (local address 127.27.174.65:39555)
0925 19:09:29.943293 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:29.943331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:29.944583 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:29.944841 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:29.944846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:29.945224 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:29.945424 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:29.945429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.946374 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.946375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:29.947196 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:29.947198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:29.947641 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:29.947642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:29.947768 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:29.948322 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:29.948330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:29.949859 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:29.951557 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:29.951581 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:29.951583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:29.951849 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:29.952086 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:29.952087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:29.952087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:29.952244 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:29.952498 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:29.952503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:29.952621 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 19:09:29.952709 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:09:29.952824 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 19:09:29.953538 29106 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34513
I20260925 19:09:29.953653 29106 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20260925 19:09:29.953843 29106 heartbeater.cc:511] Master 127.27.174.126:34513 requested a full tablet report, sending...
I20260925 19:09:29.954435 28915 ts_manager.cc:194] Registered new tserver with Master: 5a4d3e1629d34213b0ea9bec3e770f01 (127.27.174.65:40475)
I20260925 19:09:29.955132 28915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:39555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:29.970287 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:30.053030 29111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:30.053292 29111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:30.053362 29111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:30.054754 29111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:30.054816 29111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:30.054860 29111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:30.056414 29111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:30.056685 29111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:30.056910 29111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:30.061961 29119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:30.062093 29111 server_base.cc:1034] running on GCE node
W20260925 19:09:30.061972 29117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:30.062173 29116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:30.062412 29111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:30.062664 29111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:30.063828 29111 hybrid_clock.cc:648] HybridClock initialized: now 1790363370063802 us; error 50 us; skew 500 ppm
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:30.066308 29111 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:30.066743 29111 webserver.cc:460] Webserver started at http://127.27.174.66:36259/ using document root <none> and password file <none>
I20260925 19:09:30.066992 29111 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:30.067058 29111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:30.067190 29111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:30.068300 29111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "acef996e52794277bb9976343f34cc17"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "e7c4b06ac6d7ae65e0f4548436df08d6"
server_key_iv: "35adc0a66378face0afa15b8d2f4c917"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.068656 29111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "acef996e52794277bb9976343f34cc17"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "e7c4b06ac6d7ae65e0f4548436df08d6"
server_key_iv: "35adc0a66378face0afa15b8d2f4c917"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.069929 29111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:30.070598 29126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.070772 29111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:30.070868 29111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "acef996e52794277bb9976343f34cc17"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "e7c4b06ac6d7ae65e0f4548436df08d6"
server_key_iv: "35adc0a66378face0afa15b8d2f4c917"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.070948 29111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:30.109376 29111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:30.111277 29111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:30.111455 29111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:30.111756 29111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:30.112080 29111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:30.112138 29111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.112171 29111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:30.112198 29111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.117700 29111 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:38409
I20260925 19:09:30.117780 29239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:38409 every 8 connection(s)
I20260925 19:09:30.118101 29111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:30.125101 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29111
I20260925 19:09:30.125195 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:30.125396 28345 external_mini_cluster.cc:1428] Setting key cdee9a40ecfd844fcade7eae1cf522fc
I20260925 19:09:30.127667 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.118879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:41755 (local address 127.27.174.126:34513)
0925 19:09:30.118947 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.118949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.119129 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:09:30.120150 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.120151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.120161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.120233 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.120835 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.121318 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.122266 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.122447 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.122500 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.124569 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.124571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.124571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.124577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.126283 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.126685 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.126686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.126686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.126745 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.127007 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.127008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.127008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.127124 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.127201 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.127464 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.127551 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:09:30.127862 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.118840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34513 (local address 127.27.174.66:41755)
0925 19:09:30.118999 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.119041 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.120041 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.120293 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.120296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.120553 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:30.120748 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.120752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.121397 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.121397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.122188 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.122189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.122519 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.122519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.122618 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.123104 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:30.123112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:30.124486 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:30.126375 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:30.126382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:30.126384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:30.126606 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:30.126819 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:30.126820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:30.126820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:30.126932 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:30.127190 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:30.127196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:30.127333 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.127444 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.127599 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 19:09:30.128333 29240 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34513
I20260925 19:09:30.128435 29240 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:30.128626 29240 heartbeater.cc:511] Master 127.27.174.126:34513 requested a full tablet report, sending...
I20260925 19:09:30.129015 28915 ts_manager.cc:194] Registered new tserver with Master: acef996e52794277bb9976343f34cc17 (127.27.174.66:38409)
I20260925 19:09:30.129551 28915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:41755
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:30.151548 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:30.233604 29245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:30.233819 29245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:30.233843 29245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:30.235023 29245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:30.235067 29245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:30.235095 29245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:30.236646 29245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:30.236894 29245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:30.237125 29245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:30.242286 29250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:30.242300 29251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:30.242318 29253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:30.242332 29245 server_base.cc:1034] running on GCE node
I20260925 19:09:30.242726 29245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:30.242933 29245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:30.244069 29245 hybrid_clock.cc:648] HybridClock initialized: now 1790363370244062 us; error 30 us; skew 500 ppm
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:30.246691 29245 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:30.247133 29245 webserver.cc:460] Webserver started at http://127.27.174.67:35049/ using document root <none> and password file <none>
I20260925 19:09:30.247376 29245 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:30.247421 29245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:30.247555 29245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:30.248644 29245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "1af696ff36104ae7889a102de5ac6134"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "ce62a144009407145d2c250fa987800e"
server_key_iv: "cfdf48138581673bb9d85d03ae2e9c93"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.249012 29245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "1af696ff36104ae7889a102de5ac6134"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "ce62a144009407145d2c250fa987800e"
server_key_iv: "cfdf48138581673bb9d85d03ae2e9c93"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.250429 29245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:30.251274 29260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.251492 29245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:30.251595 29245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "1af696ff36104ae7889a102de5ac6134"
format_stamp: "Formatted at 2026-09-25 19:09:30 on dist-test-slave-fcdn"
server_key: "ce62a144009407145d2c250fa987800e"
server_key_iv: "cfdf48138581673bb9d85d03ae2e9c93"
server_key_version: "encryptionkey@0"
I20260925 19:09:30.251689 29245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:30.273367 29245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:30.276011 29245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:30.276165 29245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:30.276430 29245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:30.276751 29245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:30.276784 29245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.276814 29245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:30.276827 29245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:30.282518 29245 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:36751
I20260925 19:09:30.282631 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:36751 every 8 connection(s)
I20260925 19:09:30.282917 29245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:30.286697 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29245
I20260925 19:09:30.286770 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:30.286921 28345 external_mini_cluster.cc:1428] Setting key e4488b6e2abe2d3e77060f2583adaa24
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:30.292238 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.283662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37243 (local address 127.27.174.126:34513)
0925 19:09:30.283723 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.283725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.283929 (+ 204us) server_negotiation.cc:407] Connection header received
0925 19:09:30.284842 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.284843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.284852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.284904 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.285469 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.285974 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.287022 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.287341 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.287428 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.289604 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.289605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.289605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.289611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.291073 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.291436 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.291437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.291437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.291485 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.291756 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.291757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.291757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.291856 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.291921 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.292127 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.292179 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:09:30.292413 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.283648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34513 (local address 127.27.174.67:37243)
0925 19:09:30.283823 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.283858 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.284725 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.284960 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.284963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.285205 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:30.285397 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.285401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.286051 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.286052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.286918 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.286921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.287456 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.287457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.287619 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.288193 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:30.288201 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:30.289527 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:30.291147 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:30.291152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:30.291154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:30.291372 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:30.291542 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:30.291542 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:30.291542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:30.291699 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:30.291925 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:30.291929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:30.292035 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.292108 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.292219 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260925 19:09:30.292874 29374 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34513
I20260925 19:09:30.292971 29374 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:30.293148 29374 heartbeater.cc:511] Master 127.27.174.126:34513 requested a full tablet report, sending...
I20260925 19:09:30.293515 28915 ts_manager.cc:194] Registered new tserver with Master: 1af696ff36104ae7889a102de5ac6134 (127.27.174.67:36751)
I20260925 19:09:30.293884 28915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:37243
I20260925 19:09:30.298655 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:09:30.305663 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.300128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.27.174.126:34513)
0925 19:09:30.300193 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.300195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.300261 (+ 66us) server_negotiation.cc:407] Connection header received
0925 19:09:30.300295 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.300295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.300303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.300342 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.300842 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.301394 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.302152 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.302328 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.302376 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.303402 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.303403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.303403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.303409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.304599 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.304912 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.304912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.304913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.304959 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.305177 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.305179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.305179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.305324 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.305416 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.305497 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.305554 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:09:30.307878 28915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:30.309340 28915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:30.315857 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38409 (local address 127.0.0.1:43608)
0925 19:09:30.311928 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.311968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.312042 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.312346 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.312349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.312351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:30.312416 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.312417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313735 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.313735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.314649 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.314651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.315536 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.315537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.315591 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.315649 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.315751 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.315792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 19:09:30.315909 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.27.174.66:38409)
0925 19:09:30.311794 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.311797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.312025 (+ 228us) server_negotiation.cc:407] Connection header received
0925 19:09:30.312164 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.312166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.312180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.312249 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:30.312503 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313615 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.314730 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.315452 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.315530 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.315590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.315779 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.315822 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260925 19:09:30.316641 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36751 (local address 127.0.0.1:56598)
0925 19:09:30.311928 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.311968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.312042 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.312362 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.312363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.312364 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:30.312418 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.312420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313718 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.313719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.314822 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.314824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.316426 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.316427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.316470 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.316473 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.316544 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.316581 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":3,"thread_start_us":123,"threads_started":1}
I20260925 19:09:30.316669 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.27.174.67:36751)
0925 19:09:30.311956 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.311959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.312041 (+ 82us) server_negotiation.cc:407] Connection header received
0925 19:09:30.312148 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.312150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.312166 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.312249 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:30.312496 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313613 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.315666 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.316350 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.316420 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.316469 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.316559 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.316600 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260925 19:09:30.316649 29174 tablet_service.cc:1467] Processing CreateTablet for tablet c93257458dea494baa78afb8e409b35a (DEFAULT_TABLE table=test-table [id=024895f32db44b1ca397274d18540e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:30.316840 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40475 (local address 127.0.0.1:50532)
0925 19:09:30.312151 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.312159 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.312235 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.312436 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.312437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.312439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:30.312499 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.312500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313514 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.313515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.314819 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.314822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.316605 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.316606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.316673 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.316677 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.316726 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.316766 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":15,"thread_start_us":98,"threads_started":1}
I20260925 19:09:30.316987 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93257458dea494baa78afb8e409b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:30.317058 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.311764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.27.174.65:40475)
0925 19:09:30.311898 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.311901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.312235 (+ 334us) server_negotiation.cc:407] Connection header received
0925 19:09:30.312301 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.312303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.312316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.312591 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:30.312682 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.313410 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.314904 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.315576 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.316520 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.316577 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.316938 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.316983 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 19:09:30.317644 29040 tablet_service.cc:1467] Processing CreateTablet for tablet c93257458dea494baa78afb8e409b35a (DEFAULT_TABLE table=test-table [id=024895f32db44b1ca397274d18540e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:30.317866 29308 tablet_service.cc:1467] Processing CreateTablet for tablet c93257458dea494baa78afb8e409b35a (DEFAULT_TABLE table=test-table [id=024895f32db44b1ca397274d18540e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:30.317973 29040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93257458dea494baa78afb8e409b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:30.318198 29308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93257458dea494baa78afb8e409b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:30.319751 29391 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Bootstrap starting.
I20260925 19:09:30.320473 29391 tablet_bootstrap.cc:654] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:30.320971 29391 log.cc:824] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:30.321229 29393 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Bootstrap starting.
I20260925 19:09:30.321817 29393 tablet_bootstrap.cc:654] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:30.322216 29393 log.cc:824] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:30.322491 29392 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Bootstrap starting.
I20260925 19:09:30.322942 29393 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: No bootstrap required, opened a new log
I20260925 19:09:30.323005 29393 ts_tablet_manager.cc:1397] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:30.323153 29392 tablet_bootstrap.cc:654] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:30.323589 29392 log.cc:824] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:30.324291 29391 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: No bootstrap required, opened a new log
I20260925 19:09:30.324378 29391 ts_tablet_manager.cc:1397] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 19:09:30.324393 29392 tablet_bootstrap.cc:492] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: No bootstrap required, opened a new log
I20260925 19:09:30.324447 29392 ts_tablet_manager.cc:1397] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:09:30.324779 29393 raft_consensus.cc:348] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.324951 29393 raft_consensus.cc:374] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:30.324986 29393 raft_consensus.cc:729] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af696ff36104ae7889a102de5ac6134, State: Initialized, Role: FOLLOWER
I20260925 19:09:30.325126 29393 consensus_queue.cc:260] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.325374 29393 ts_tablet_manager.cc:1428] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:09:30.325492 29374 heartbeater.cc:503] Master 127.27.174.126:34513 was elected leader, sending a full tablet report...
I20260925 19:09:30.326179 29391 raft_consensus.cc:348] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.326375 29391 raft_consensus.cc:374] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:30.326423 29391 raft_consensus.cc:729] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acef996e52794277bb9976343f34cc17, State: Initialized, Role: FOLLOWER
I20260925 19:09:30.326535 29391 consensus_queue.cc:260] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.326468 29392 raft_consensus.cc:348] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.326642 29392 raft_consensus.cc:374] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:30.326689 29392 raft_consensus.cc:729] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4d3e1629d34213b0ea9bec3e770f01, State: Initialized, Role: FOLLOWER
I20260925 19:09:30.326789 29391 ts_tablet_manager.cc:1428] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:09:30.326792 29392 consensus_queue.cc:260] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.326927 29240 heartbeater.cc:503] Master 127.27.174.126:34513 was elected leader, sending a full tablet report...
I20260925 19:09:30.327024 29106 heartbeater.cc:503] Master 127.27.174.126:34513 was elected leader, sending a full tablet report...
I20260925 19:09:30.327028 29392 ts_tablet_manager.cc:1428] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 19:09:30.368678 29241 tablet.cc:2369] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:09:30.443181 29107 tablet.cc:2369] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:30.451805 29399 raft_consensus.cc:482] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:30.451920 29399 raft_consensus.cc:504] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.452286 29399 leader_election.cc:288] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acef996e52794277bb9976343f34cc17 (127.27.174.66:38409), 1af696ff36104ae7889a102de5ac6134 (127.27.174.67:36751)
I20260925 19:09:30.455534 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.452561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:56125 (local address 127.27.174.66:38409)
0925 19:09:30.452609 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.452611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.452756 (+ 145us) server_negotiation.cc:407] Connection header received
0925 19:09:30.452827 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.452828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.452837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.452913 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:30.453076 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.453534 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.454723 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.455193 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.455275 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.455285 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.455430 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.455465 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:09:30.455552 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.452495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38409 (local address 127.27.174.65:56125)
0925 19:09:30.452666 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.452677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.452768 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.452914 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.452914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.452916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:30.452999 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.453000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.453617 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.453618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.454575 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.454577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.455273 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.455274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.455345 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.455349 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.455424 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.455465 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":56,"thread_start_us":63,"threads_started":1}
I20260925 19:09:30.455672 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.452490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36751 (local address 127.27.174.65:47705)
0925 19:09:30.452582 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:30.452596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:30.452675 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:30.452868 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:30.452869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:30.452870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:30.452953 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.452955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.453656 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.453657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.454656 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:30.454658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.455312 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.455313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:30.455399 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.455408 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:09:30.455477 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:09:30.455515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":46}
I20260925 19:09:30.455915 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.452545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47705 (local address 127.27.174.67:36751)
0925 19:09:30.452608 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.452611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.452661 (+ 50us) server_negotiation.cc:407] Connection header received
0925 19:09:30.452788 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.452789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.452797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.452865 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:30.453024 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.453574 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.454728 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.455233 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.455314 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.455323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.455577 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.455847 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:09:30.456020 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93257458dea494baa78afb8e409b35a" candidate_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acef996e52794277bb9976343f34cc17" is_pre_election: true
I20260925 19:09:30.456224 29194 raft_consensus.cc:2454] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4d3e1629d34213b0ea9bec3e770f01 in term 0.
I20260925 19:09:30.456190 29328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93257458dea494baa78afb8e409b35a" candidate_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af696ff36104ae7889a102de5ac6134" is_pre_election: true
I20260925 19:09:30.456358 29328 raft_consensus.cc:2454] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4d3e1629d34213b0ea9bec3e770f01 in term 0.
I20260925 19:09:30.456610 28993 leader_election.cc:302] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af696ff36104ae7889a102de5ac6134, 5a4d3e1629d34213b0ea9bec3e770f01; no voters:
I20260925 19:09:30.456810 29399 raft_consensus.cc:2781] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:30.456851 29399 raft_consensus.cc:482] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:30.456894 29399 raft_consensus.cc:3037] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:30.457566 29399 raft_consensus.cc:504] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.457742 29399 leader_election.cc:288] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [CANDIDATE]: Term 1 election: Requested vote from peers acef996e52794277bb9976343f34cc17 (127.27.174.66:38409), 1af696ff36104ae7889a102de5ac6134 (127.27.174.67:36751)
I20260925 19:09:30.457876 29328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93257458dea494baa78afb8e409b35a" candidate_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af696ff36104ae7889a102de5ac6134"
I20260925 19:09:30.457881 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93257458dea494baa78afb8e409b35a" candidate_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acef996e52794277bb9976343f34cc17"
I20260925 19:09:30.457952 29328 raft_consensus.cc:3037] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:30.457978 29194 raft_consensus.cc:3037] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:30.458832 29328 raft_consensus.cc:2454] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4d3e1629d34213b0ea9bec3e770f01 in term 1.
I20260925 19:09:30.458863 29194 raft_consensus.cc:2454] T c93257458dea494baa78afb8e409b35a P acef996e52794277bb9976343f34cc17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4d3e1629d34213b0ea9bec3e770f01 in term 1.
I20260925 19:09:30.459008 28993 leader_election.cc:302] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af696ff36104ae7889a102de5ac6134, 5a4d3e1629d34213b0ea9bec3e770f01; no voters:
I20260925 19:09:30.459111 29399 raft_consensus.cc:2781] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:30.459300 29399 raft_consensus.cc:686] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4d3e1629d34213b0ea9bec3e770f01, State: Running, Role: LEADER
I20260925 19:09:30.459403 29399 consensus_queue.cc:237] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } }
I20260925 19:09:30.460192 28915 catalog_manager.cc:5495] T c93257458dea494baa78afb8e409b35a P 5a4d3e1629d34213b0ea9bec3e770f01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4d3e1629d34213b0ea9bec3e770f01 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acef996e52794277bb9976343f34cc17" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af696ff36104ae7889a102de5ac6134" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4d3e1629d34213b0ea9bec3e770f01" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40475 } health_report { overall_health: HEALTHY } } }
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:30.503866 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.496190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.27.174.65:40475)
0925 19:09:30.496270 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.496271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.496296 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:09:30.496369 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.496370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.496380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.496435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.497020 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.497527 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.498497 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.498687 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.498752 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.501294 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.501305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.501305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.501312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.502790 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.503119 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.503123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.503123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.503189 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.503356 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.503357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.503357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.503491 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.503657 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.503717 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.503780 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:30.526455 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.519243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.27.174.65:40475)
0925 19:09:30.519306 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.519309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.519493 (+ 184us) server_negotiation.cc:407] Connection header received
0925 19:09:30.519522 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.519523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.519533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.519577 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.520218 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.520742 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.521449 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.521676 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.521736 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.524139 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.524140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.524140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.524146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.525303 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.525669 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.525670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.525670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.525725 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.525983 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.525984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.525984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.526084 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.526161 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.526319 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.526390 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 19:09:30.533454 29375 tablet.cc:2369] T c93257458dea494baa78afb8e409b35a P 1af696ff36104ae7889a102de5ac6134: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[28870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:30.548172 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:30.541312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.27.174.65:40475)
0925 19:09:30.541384 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:30.541386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:30.541505 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:09:30.541583 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:30.541584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:30.541594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:30.541666 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:30.542232 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.542705 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.543439 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:30.543682 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:30.543767 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:30.545916 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:30.545917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:30.545918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:30.545924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:30.547162 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.547471 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.547472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.547473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.547519 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:30.547786 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:30.547787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:30.547787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:30.547879 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:30.547952 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:30.548055 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:09:30.548109 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 19:09:30.548655 29020 tablet_service.cc:2908] Rejecting scan request for tablet c93257458dea494baa78afb8e409b35a: Uninitialized: clean time has not yet been initialized
I20260925 19:09:30.549787 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28977
I20260925 19:09:30.554845 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29111
I20260925 19:09:30.559507 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29245
I20260925 19:09:30.563877 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 28883
2026-09-25T19:09:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2096 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[29415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[29415](info): set up 2 sockets
Sep 25 19:09:30 dist-test-slave-fcdn krb5kdc[29415](info): commencing operation
krb5kdc: starting...
W20260925 19:09:31.595297 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:31 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:31.707496 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38805
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:31.787868 29428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:31.788218 29428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:31.788252 29428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:31.789904 29428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:31.789959 29428 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:09:31.789978 29428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:31.789996 29428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:31.790015 29428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:31.790033 29428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:31.791955 29428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:31.792148 29428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:31.792305 29428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:31.797061 29434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:31.797106 29433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:31.797070 29436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:31.797444 29428 server_base.cc:1034] running on GCE node
I20260925 19:09:31.797655 29428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:31.797924 29428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:31.799077 29428 hybrid_clock.cc:648] HybridClock initialized: now 1790363371799053 us; error 39 us; skew 500 ppm
Sep 25 19:09:31 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:31.806006 29428 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:31.806468 29428 webserver.cc:460] Webserver started at http://127.27.174.126:38019/ using document root <none> and password file <none>
I20260925 19:09:31.806696 29428 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:31.806759 29428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:31.806916 29428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:31.808043 29428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "6208e4c743ad474baaba96b15a936121"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "53daabe446481d86038ce44b89b45c99"
server_key_iv: "95f8c81b11b6830fd915d8870201e8fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.808400 29428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "6208e4c743ad474baaba96b15a936121"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "53daabe446481d86038ce44b89b45c99"
server_key_iv: "95f8c81b11b6830fd915d8870201e8fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.809679 29428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:31.810365 29443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:31.810557 29428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:31.810672 29428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "6208e4c743ad474baaba96b15a936121"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "53daabe446481d86038ce44b89b45c99"
server_key_iv: "95f8c81b11b6830fd915d8870201e8fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.810748 29428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:31.825021 29428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:31.826767 29428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:31.826947 29428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:31.831710 29428 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38805
I20260925 19:09:31.831768 29505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38805 every 8 connection(s)
I20260925 19:09:31.832089 29428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:31.832719 29506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:31.834908 29506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Bootstrap starting.
I20260925 19:09:31.835474 29506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:31.835778 29506 log.cc:824] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:31.836438 29506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: No bootstrap required, opened a new log
I20260925 19:09:31.838469 29506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } }
I20260925 19:09:31.838608 29506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:31.838629 29506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6208e4c743ad474baaba96b15a936121, State: Initialized, Role: FOLLOWER
I20260925 19:09:31.838754 29506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } }
I20260925 19:09:31.838835 29506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:31.838861 29506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:31.838900 29506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:31.839429 29506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } }
I20260925 19:09:31.839535 29506 leader_election.cc:302] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6208e4c743ad474baaba96b15a936121; no voters:
I20260925 19:09:31.839746 29506 leader_election.cc:288] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:31.839793 29509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:31.839977 29506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:31.839985 29509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [term 1 LEADER]: Becoming Leader. State: Replica: 6208e4c743ad474baaba96b15a936121, State: Running, Role: LEADER
I20260925 19:09:31.840103 29509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } }
I20260925 19:09:31.840529 29510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6208e4c743ad474baaba96b15a936121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } } }
I20260925 19:09:31.840624 29510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:31.840598 29511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6208e4c743ad474baaba96b15a936121. Latest consensus state: current_term: 1 leader_uuid: "6208e4c743ad474baaba96b15a936121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6208e4c743ad474baaba96b15a936121" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38805 } } }
I20260925 19:09:31.840680 29511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:31.840853 29518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:31.841327 29518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:31.842034 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29428
I20260925 19:09:31.842129 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:31.842274 28345 external_mini_cluster.cc:1428] Setting key 79f081ce6c6237ac29a6ce61a39e76b3
I20260925 19:09:31.843238 29518 catalog_manager.cc:1347] Generated new cluster ID: 36835673823849088ef03499410d1aa6
I20260925 19:09:31.843302 29518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:31 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:31.851867 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:31.842702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.27.174.126:38805)
0925 19:09:31.843309 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:31.843318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:31.843359 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:09:31.843463 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:31.843468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:31.843508 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:31.843671 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:31.844263 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:31.845170 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:31.845919 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:31.846320 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:31.846441 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:31.848708 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:31.848711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:31.848712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:31.848722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:31.850342 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:31.850702 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:31.850705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:31.850706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:31.850804 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:31.851018 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:31.851019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:31.851020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:31.851249 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:31.851417 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:31.851520 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:09:31.851673 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":89,"threads_started":1}
I20260925 19:09:31.863313 29518 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:31.863888 29518 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:31.869156 29518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Generated new TSK 0
I20260925 19:09:31.869339 29518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:31.896311 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38805
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:31.976576 29532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:31.976809 29532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:31.976850 29532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:31.978012 29532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:31.978072 29532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:31.978107 29532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:31.978164 29532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:31.979645 29532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:31.979910 29532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:31.980144 29532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:31.985091 29540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:31.985301 29532 server_base.cc:1034] running on GCE node
W20260925 19:09:31.985116 29538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:31.985103 29537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:31.985597 29532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:31.985857 29532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:31.987031 29532 hybrid_clock.cc:648] HybridClock initialized: now 1790363371987022 us; error 39 us; skew 500 ppm
Sep 25 19:09:31 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:31.989534 29532 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:31.989996 29532 webserver.cc:460] Webserver started at http://127.27.174.65:34415/ using document root <none> and password file <none>
I20260925 19:09:31.990262 29532 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:31.990345 29532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:31.990492 29532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:31.991549 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "d1d06dc02b0a4c24aaea193901cb501b"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "e4df699d4d0abe42c87eb95b7ad4b5ab"
server_key_iv: "301f2985c8df2a5e11b6f568d9641d07"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.991981 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "d1d06dc02b0a4c24aaea193901cb501b"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "e4df699d4d0abe42c87eb95b7ad4b5ab"
server_key_iv: "301f2985c8df2a5e11b6f568d9641d07"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.993290 29532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:31.994019 29547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:31.994194 29532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:31.994297 29532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "d1d06dc02b0a4c24aaea193901cb501b"
format_stamp: "Formatted at 2026-09-25 19:09:31 on dist-test-slave-fcdn"
server_key: "e4df699d4d0abe42c87eb95b7ad4b5ab"
server_key_iv: "301f2985c8df2a5e11b6f568d9641d07"
server_key_version: "encryptionkey@0"
I20260925 19:09:31.994371 29532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:32.002701 29532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:32.005116 29532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:32.005285 29532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:32.005812 29532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:32.005878 29532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.005939 29532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:32.005968 29532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:32.013515 29532 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41591
I20260925 19:09:32.013887 29662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41591 every 8 connection(s)
I20260925 19:09:32.013963 29532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:32.017143 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.006479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33941 (local address 127.27.174.126:38805)
0925 19:09:32.006556 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.006559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.006772 (+ 213us) server_negotiation.cc:407] Connection header received
0925 19:09:32.007433 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.007435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.007453 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.007535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.008258 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.009113 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.010359 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.010720 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.010802 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.013516 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.013517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.013517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.013525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.015503 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.016028 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.016030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.016030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.016104 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.016410 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.016411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.016412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.016547 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.016634 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.016940 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.017042 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:09:32.017323 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.006462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.65:33941)
0925 19:09:32.006648 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.006691 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.007278 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.007648 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.007653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.007927 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.008131 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.008136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.009232 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.009233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.010273 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.010277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.010941 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.010944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.011128 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.011755 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.011764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.013407 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.015643 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.015650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.015653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.015921 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.016198 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.016199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.016199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.016326 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.016622 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.016629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.016789 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.016901 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.017070 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260925 19:09:32.017879 29663 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38805
I20260925 19:09:32.017992 29663 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:32.018187 29663 heartbeater.cc:511] Master 127.27.174.126:38805 requested a full tablet report, sending...
I20260925 19:09:32.018759 29459 ts_manager.cc:194] Registered new tserver with Master: d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591)
I20260925 19:09:32.019712 29459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:33941
I20260925 19:09:32.021174 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29532
I20260925 19:09:32.021277 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:32.021472 28345 external_mini_cluster.cc:1428] Setting key cef543b767209468e254937150fe9f81
I20260925 19:09:32.025306 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.018969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34403 (local address 127.27.174.126:38805)
0925 19:09:32.019019 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.019021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.019174 (+ 153us) server_negotiation.cc:407] Connection header received
0925 19:09:32.019199 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.019200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.019211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.019265 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.019558 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.020323 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.021026 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.021369 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.021455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.022482 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.022484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.022484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.022490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.024149 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.024496 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.024497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.024497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.024556 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.024826 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.024826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.024827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.024939 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.024991 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.025186 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.025248 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:09:32.025449 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.018920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.65:34403)
0925 19:09:32.019075 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.019083 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.019133 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.019277 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.019277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.019434 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.019490 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.019492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.020423 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.020424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.020945 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.020946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.021450 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.021451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.021506 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.021970 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.021974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.022392 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.024242 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.024243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.024244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.024417 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.024631 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.024632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.024632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.024754 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.025008 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.025009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.025079 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.025162 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.025287 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:32.046214 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38805
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:32.126050 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:32.126245 29672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:32.126267 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:32.127384 29672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:32.127418 29672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:32.127437 29672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:32.127453 29672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:32.128790 29672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:32.129006 29672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:32.129169 29672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:09:32.134060 29672 server_base.cc:1034] running on GCE node
W20260925 19:09:32.134016 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:32.134025 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:32.134021 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:32.134459 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:32.134725 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:32.135891 29672 hybrid_clock.cc:648] HybridClock initialized: now 1790363372135877 us; error 27 us; skew 500 ppm
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:32.138303 29672 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:32.138891 29672 webserver.cc:460] Webserver started at http://127.27.174.66:36267/ using document root <none> and password file <none>
I20260925 19:09:32.139128 29672 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:32.139174 29672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:32.139262 29672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:32.140466 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "430e354af5a54d3bb2f98528ff9f90fd"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "9568c4059a96e01660a023a45f98795d"
server_key_iv: "6ec688238535178136981dade212ea69"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.140847 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "430e354af5a54d3bb2f98528ff9f90fd"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "9568c4059a96e01660a023a45f98795d"
server_key_iv: "6ec688238535178136981dade212ea69"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.142180 29672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:32.142859 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.143044 29672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:32.143133 29672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "430e354af5a54d3bb2f98528ff9f90fd"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "9568c4059a96e01660a023a45f98795d"
server_key_iv: "6ec688238535178136981dade212ea69"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.143203 29672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:32.169692 29672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:32.172185 29672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:32.172358 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:32.173126 29672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:32.173182 29672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.173241 29672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:32.173273 29672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:32.180176 29672 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:38731
I20260925 19:09:32.180253 29802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:38731 every 8 connection(s)
I20260925 19:09:32.180600 29672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:32.181315 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29672
I20260925 19:09:32.181417 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:32.181654 28345 external_mini_cluster.cc:1428] Setting key bf42ee2fb0bcca3c4a8a098e75b25377
I20260925 19:09:32.183406 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.173550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:33279 (local address 127.27.174.126:38805)
0925 19:09:32.173614 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.173616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.173811 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:09:32.174363 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.174365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.174377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.174433 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.175285 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.176127 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.177520 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.177817 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.177892 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.180180 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.180182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.180182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.180190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.181980 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.182453 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.182454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.182455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.182511 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.182767 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.182767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.182768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.182883 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.182959 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.183226 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.183304 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:09:32.183566 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.173541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.66:33279)
0925 19:09:32.173691 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.173727 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.174248 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.174502 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.174506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.174884 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.175162 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.175169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.176209 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.176210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.177415 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.177418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.177976 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.177978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.178080 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.178560 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.178568 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.180073 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.182059 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.182064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.182066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.182368 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.182585 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.182587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.182587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.182690 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.182950 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.182956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.183089 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.183189 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.183338 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260925 19:09:32.184861 29803 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38805
I20260925 19:09:32.184945 29803 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:32.185115 29803 heartbeater.cc:511] Master 127.27.174.126:38805 requested a full tablet report, sending...
I20260925 19:09:32.185516 29460 ts_manager.cc:194] Registered new tserver with Master: 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731)
I20260925 19:09:32.186080 29460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:33279
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20260925 19:09:32.190740 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.185160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:43181 (local address 127.27.174.126:38805)
0925 19:09:32.185195 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.185197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.185204 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:32.185231 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.185231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.185241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.185295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.185686 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.186259 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.187028 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.187239 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.187309 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.188272 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.188273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.188274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.188280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.189609 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.189945 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.189946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.189946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.189996 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.190193 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.190194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.190194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.190351 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.190421 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.190581 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.190651 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:09:32.190752 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.184975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.66:43181)
0925 19:09:32.185095 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.185105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.185168 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.185298 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.185299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.185518 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.185592 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.185593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.186363 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.186364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.186917 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.186919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.187362 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.187362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.187446 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.187915 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.187919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.188183 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.189701 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.189701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.189702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.189872 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.190068 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.190068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.190069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.190123 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.190423 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.190424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.190475 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.190558 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.190652 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:32.206947 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38805
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:32.286399 29812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:32.286624 29812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:32.286659 29812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:32.287819 29812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:32.287887 29812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:32.287920 29812 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:32.287961 29812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:32.289372 29812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:32.289634 29812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:32.289856 29812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:32.294770 29818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:32.294791 29820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:32.294795 29817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:32.294993 29812 server_base.cc:1034] running on GCE node
I20260925 19:09:32.295192 29812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:32.295432 29812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:32.296581 29812 hybrid_clock.cc:648] HybridClock initialized: now 1790363372296566 us; error 30 us; skew 500 ppm
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:32.299168 29812 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:32.299688 29812 webserver.cc:460] Webserver started at http://127.27.174.67:44559/ using document root <none> and password file <none>
I20260925 19:09:32.299923 29812 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:32.299985 29812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:32.300129 29812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:32.301113 29812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "2454883df6254294a5c187e36701594f"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "ee2a184025397eb03ae5d081958cc416"
server_key_iv: "e20f961357f340c77ac17a7beedf167d"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.301453 29812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "2454883df6254294a5c187e36701594f"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "ee2a184025397eb03ae5d081958cc416"
server_key_iv: "e20f961357f340c77ac17a7beedf167d"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.302675 29812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:32.303458 29827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.303712 29812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:32.303781 29812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2454883df6254294a5c187e36701594f"
format_stamp: "Formatted at 2026-09-25 19:09:32 on dist-test-slave-fcdn"
server_key: "ee2a184025397eb03ae5d081958cc416"
server_key_iv: "e20f961357f340c77ac17a7beedf167d"
server_key_version: "encryptionkey@0"
I20260925 19:09:32.303872 29812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:32.320588 29812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:32.322865 29812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:32.323010 29812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:32.323539 29812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:32.323596 29812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.323664 29812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:32.323685 29812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:32.330967 29812 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:43563
I20260925 19:09:32.331029 29942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:43563 every 8 connection(s)
I20260925 19:09:32.331323 29812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:32.331501 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 29812
I20260925 19:09:32.331580 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:32.331784 28345 external_mini_cluster.cc:1428] Setting key c400326a0f13549a10cffaabbfa6ee3c
I20260925 19:09:32.332365 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.323914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:48297 (local address 127.27.174.126:38805)
0925 19:09:32.323969 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.323971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.324269 (+ 298us) server_negotiation.cc:407] Connection header received
0925 19:09:32.324708 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.324710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.324722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.324815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.325366 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.325917 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.326895 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.327099 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.327183 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.329362 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.329363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.329364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.329371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.330854 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.331292 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.331293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.331294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.331349 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.331655 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.331656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.331657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.331789 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.331876 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.332178 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.332255 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:09:32.332551 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.323927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.67:48297)
0925 19:09:32.324125 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.324164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.324613 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.324862 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.324865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.325074 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.325255 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.325258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.326010 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.326011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.326798 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.326800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.327192 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.327193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.327307 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.327795 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.327802 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.329261 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.330922 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.330926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.330928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.331204 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.331425 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.331425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.331426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.331556 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.331867 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.331870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.332029 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.332142 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.332312 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":136,"threads_started":1}
I20260925 19:09:32.333137 29943 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38805
I20260925 19:09:32.333216 29943 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:32.333379 29943 heartbeater.cc:511] Master 127.27.174.126:38805 requested a full tablet report, sending...
I20260925 19:09:32.333783 29460 ts_manager.cc:194] Registered new tserver with Master: 2454883df6254294a5c187e36701594f (127.27.174.67:43563)
I20260925 19:09:32.334300 29460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:48297
I20260925 19:09:32.339506 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.334168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.27.174.67:33007)
0925 19:09:32.334279 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.334290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.334347 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.334472 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.334473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.334662 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.334714 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.334716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.335346 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.335347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.335909 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.335911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.336288 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.336288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.336362 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.336752 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.336759 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.337027 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.338510 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.338511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.338511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.338673 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.338864 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.338865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.338866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.338919 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.339180 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.339181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.339231 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.339346 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.339443 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260925 19:09:32.339519 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.334232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33007 (local address 127.27.174.126:38805)
0925 19:09:32.334274 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.334276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.334337 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:09:32.334403 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.334403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.334413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.334467 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.334799 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.335258 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.336001 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.336191 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.336286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.337130 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.337132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.337132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.337138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.338416 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.338744 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.338745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.338746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.338795 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.338990 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.338991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.338992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.339109 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.339175 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.339406 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.339467 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:32.343278 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:32.364333 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.357931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.27.174.126:38805)
0925 19:09:32.357998 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.358012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.358021 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:32.358046 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.358047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.358063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.358124 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.358616 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.359078 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.359846 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.360026 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.360112 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.362100 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.362101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.362101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.362109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.363189 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.363522 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.363523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.363524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.363579 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.363817 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.363819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.363819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.363928 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.364053 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.364156 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.364234 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:09:32.365943 29460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:32.367399 29460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:32.373636 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.27.174.65:41591)
0925 19:09:32.370164 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.370169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.370179 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:32.370213 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.370214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.370228 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.370296 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.370444 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371541 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.372704 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373239 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373341 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373400 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.373513 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.373550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
I20260925 19:09:32.373625 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41591 (local address 127.0.0.1:52834)
0925 19:09:32.369968 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.369976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.370016 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.370306 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.370307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.370309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.370362 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.370364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371654 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.371655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.372622 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.372624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373344 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.373417 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373430 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.373500 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.373548 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 19:09:32.373863 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38731 (local address 127.0.0.1:33556)
0925 19:09:32.369967 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.369977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.370032 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.370225 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.370228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.370230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.370297 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.370298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371264 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.371265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.372528 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.372530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373446 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.373515 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373518 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.373569 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.373610 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":57,"threads_started":1}
I20260925 19:09:32.374078 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.27.174.66:38731)
0925 19:09:32.369913 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.369918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.370040 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:09:32.370091 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.370093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.370109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.370203 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.370382 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371178 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.372613 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373278 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.373665 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.373868 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 19:09:32.374145 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:43563 (local address 127.0.0.1:56722)
0925 19:09:32.369769 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.369812 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.369878 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.370578 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.370579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.370581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.370616 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.370618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371755 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.371756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.372795 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.372797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373672 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.373865 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373869 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.374030 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.374068 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260925 19:09:32.374384 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.369768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.27.174.67:43563)
0925 19:09:32.370418 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.370422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.370431 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:32.370465 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.370468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.370484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.370581 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.370677 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.371681 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.372870 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.373541 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.373597 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.373640 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.374233 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.374272 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":119,"threads_started":1}
I20260925 19:09:32.374347 29597 tablet_service.cc:1467] Processing CreateTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 (DEFAULT_TABLE table=test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.374715 29597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7591ffff5c4db0943333b4d592bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.374792 29737 tablet_service.cc:1467] Processing CreateTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 (DEFAULT_TABLE table=test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.374907 29877 tablet_service.cc:1467] Processing CreateTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 (DEFAULT_TABLE table=test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.375111 29737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7591ffff5c4db0943333b4d592bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.375165 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7591ffff5c4db0943333b4d592bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.377642 29963 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Bootstrap starting.
I20260925 19:09:32.378204 29964 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Bootstrap starting.
I20260925 19:09:32.378340 29963 tablet_bootstrap.cc:654] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.378700 29963 log.cc:824] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:32.379115 29964 tablet_bootstrap.cc:654] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.379407 29964 log.cc:824] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:32.379539 29963 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: No bootstrap required, opened a new log
I20260925 19:09:32.379652 29963 ts_tablet_manager.cc:1397] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:32.379763 29965 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Bootstrap starting.
I20260925 19:09:32.380425 29965 tablet_bootstrap.cc:654] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.380736 29965 log.cc:824] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:32.381476 29965 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: No bootstrap required, opened a new log
I20260925 19:09:32.381543 29965 ts_tablet_manager.cc:1397] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:32.381810 29963 raft_consensus.cc:348] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.381980 29963 raft_consensus.cc:374] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.382005 29963 raft_consensus.cc:729] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d06dc02b0a4c24aaea193901cb501b, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.382131 29963 consensus_queue.cc:260] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.382388 29963 ts_tablet_manager.cc:1428] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:32.382501 29663 heartbeater.cc:503] Master 127.27.174.126:38805 was elected leader, sending a full tablet report...
I20260925 19:09:32.383296 29965 raft_consensus.cc:348] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.383433 29965 raft_consensus.cc:374] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.383476 29965 raft_consensus.cc:729] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e354af5a54d3bb2f98528ff9f90fd, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.383575 29965 consensus_queue.cc:260] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.383890 29965 ts_tablet_manager.cc:1428] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:09:32.383945 29803 heartbeater.cc:503] Master 127.27.174.126:38805 was elected leader, sending a full tablet report...
I20260925 19:09:32.384006 29964 tablet_bootstrap.cc:492] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: No bootstrap required, opened a new log
I20260925 19:09:32.384071 29964 ts_tablet_manager.cc:1397] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 19:09:32.385795 29964 raft_consensus.cc:348] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.385915 29964 raft_consensus.cc:374] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.385936 29964 raft_consensus.cc:729] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2454883df6254294a5c187e36701594f, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.386009 29964 consensus_queue.cc:260] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.386183 29964 ts_tablet_manager.cc:1428] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:32.386262 29943 heartbeater.cc:503] Master 127.27.174.126:38805 was elected leader, sending a full tablet report...
W20260925 19:09:32.431350 29804 tablet.cc:2369] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:32.449563 29971 raft_consensus.cc:482] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:32.449677 29971 raft_consensus.cc:504] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.449996 29971 leader_election.cc:288] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591)
I20260925 19:09:32.453927 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.450209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33177 (local address 127.27.174.66:38731)
0925 19:09:32.450263 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.450265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.450649 (+ 384us) server_negotiation.cc:407] Connection header received
0925 19:09:32.450675 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.450676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.450685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.450732 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.450856 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.451546 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.452947 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.453502 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.453581 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.453589 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.453819 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.453854 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:09:32.453931 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.450260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:54845 (local address 127.27.174.65:41591)
0925 19:09:32.450333 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.450335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.450344 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:32.450371 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.450372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.450382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.450445 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.450647 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.451462 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.452947 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.453502 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.453581 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.453589 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.453819 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.453860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:09:32.453958 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.450160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41591 (local address 127.27.174.67:54845)
0925 19:09:32.450201 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.450213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.450255 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.450491 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.450492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.450494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.450569 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.450571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.451545 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.451546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.452854 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.452857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.453623 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.453624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.453734 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.453737 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.453791 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.453844 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":37}
I20260925 19:09:32.453964 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.450163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38731 (local address 127.27.174.67:33177)
0925 19:09:32.450535 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.450547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.450613 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.450742 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.450742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.450743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.450795 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.450796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.451666 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.451667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.452854 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.452857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.453623 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.453624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.453724 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.453729 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.453791 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.453844 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":205,"thread_start_us":122,"threads_started":1}
I20260925 19:09:32.454339 29617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7591ffff5c4db0943333b4d592bcc3" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d06dc02b0a4c24aaea193901cb501b" is_pre_election: true
I20260925 19:09:32.454396 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7591ffff5c4db0943333b4d592bcc3" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd" is_pre_election: true
I20260925 19:09:32.454509 29617 raft_consensus.cc:2454] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2454883df6254294a5c187e36701594f in term 0.
I20260925 19:09:32.454509 29757 raft_consensus.cc:2454] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2454883df6254294a5c187e36701594f in term 0.
I20260925 19:09:32.454737 29831 leader_election.cc:302] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2454883df6254294a5c187e36701594f, 430e354af5a54d3bb2f98528ff9f90fd; no voters:
I20260925 19:09:32.454895 29971 raft_consensus.cc:2781] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:32.454952 29971 raft_consensus.cc:482] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:32.454975 29971 raft_consensus.cc:3037] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.455688 29971 raft_consensus.cc:504] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.455834 29971 leader_election.cc:288] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 election: Requested vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591)
I20260925 19:09:32.455986 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7591ffff5c4db0943333b4d592bcc3" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd"
I20260925 19:09:32.455996 29617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7591ffff5c4db0943333b4d592bcc3" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d06dc02b0a4c24aaea193901cb501b"
I20260925 19:09:32.456064 29617 raft_consensus.cc:3037] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.456064 29757 raft_consensus.cc:3037] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.456930 29617 raft_consensus.cc:2454] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2454883df6254294a5c187e36701594f in term 1.
I20260925 19:09:32.456930 29757 raft_consensus.cc:2454] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2454883df6254294a5c187e36701594f in term 1.
I20260925 19:09:32.457087 29831 leader_election.cc:302] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2454883df6254294a5c187e36701594f, 430e354af5a54d3bb2f98528ff9f90fd; no voters:
I20260925 19:09:32.457182 29971 raft_consensus.cc:2781] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:32.457343 29971 raft_consensus.cc:686] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 1 LEADER]: Becoming Leader. State: Replica: 2454883df6254294a5c187e36701594f, State: Running, Role: LEADER
I20260925 19:09:32.457442 29971 consensus_queue.cc:237] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } }
I20260925 19:09:32.458317 29460 catalog_manager.cc:5495] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2454883df6254294a5c187e36701594f (127.27.174.67). New cstate: current_term: 1 leader_uuid: "2454883df6254294a5c187e36701594f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:32.470247 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.467502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.27.174.67:43563)
0925 19:09:32.467543 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.467545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.467567 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:09:32.467695 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.467696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.467721 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.467794 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:32.467976 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.468445 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.469262 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.469455 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.469529 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.469752 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.469847 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.469932 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.469984 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.470002 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:32.471930 29757 raft_consensus.cc:1261] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Refusing update from remote peer 2454883df6254294a5c187e36701594f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.471930 29617 raft_consensus.cc:1261] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Refusing update from remote peer 2454883df6254294a5c187e36701594f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.472246 29973 consensus_queue.cc:1035] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.472417 29971 consensus_queue.cc:1035] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [LEADER]: Connected to new peer: Peer: permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.475044 29977 mvcc.cc:205] Tried to move back new op lower bound from 7333328373642326016 to 7333328373586403328. Current Snapshot: MvccSnapshot[applied={T|T < 7333328373642326016}]
I20260925 19:09:32.475121 29976 mvcc.cc:205] Tried to move back new op lower bound from 7333328373642326016 to 7333328373586403328. Current Snapshot: MvccSnapshot[applied={T|T < 7333328373642326016}]
I20260925 19:09:32.476475 29979 mvcc.cc:205] Tried to move back new op lower bound from 7333328373642326016 to 7333328373586403328. Current Snapshot: MvccSnapshot[applied={T|T < 7333328373642326016}]
I20260925 19:09:32.478446 29459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:32.478542 29459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:32.479277 29459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Sending DeleteTablet for 3 replicas of tablet aa7591ffff5c4db0943333b4d592bcc3
I20260925 19:09:32.479557 29877 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56722
I20260925 19:09:32.479581 29737 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33556
I20260925 19:09:32.479602 29597 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7591ffff5c4db0943333b4d592bcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52834
I20260925 19:09:32.479861 29987 tablet_replica.cc:330] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: stopping tablet replica
I20260925 19:09:32.479959 29987 raft_consensus.cc:2229] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:32.479895 29459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:32.480036 29459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:32.480067 29987 raft_consensus.cc:2258] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.480314 29986 tablet_replica.cc:330] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: stopping tablet replica
I20260925 19:09:32.480409 29986 raft_consensus.cc:2229] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:32.480507 29987 ts_tablet_manager.cc:1905] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.481159 29986 raft_consensus.cc:2258] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.481534 29737 tablet_service.cc:1467] Processing CreateTablet for tablet 3a551b13451448beb6f86f876946abc1 (DEFAULT_TABLE table=test-table [id=c5728d6b7e484fafb776f671287b390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.481660 29737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a551b13451448beb6f86f876946abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.481846 29986 ts_tablet_manager.cc:1905] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.482667 29987 ts_tablet_manager.cc:1918] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:32.482726 29987 log.cc:1196] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/aa7591ffff5c4db0943333b4d592bcc3
I20260925 19:09:32.482965 29965 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Bootstrap starting.
I20260925 19:09:32.482990 29987 ts_tablet_manager.cc:1939] T aa7591ffff5c4db0943333b4d592bcc3 P d1d06dc02b0a4c24aaea193901cb501b: Deleting consensus metadata
I20260925 19:09:32.483345 29965 tablet_bootstrap.cc:654] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.483697 29445 catalog_manager.cc:4841] TS d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591): tablet aa7591ffff5c4db0943333b4d592bcc3 (table test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]) successfully deleted
I20260925 19:09:32.481534 29985 tablet_replica.cc:330] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: stopping tablet replica
I20260925 19:09:32.483800 29877 tablet_service.cc:1467] Processing CreateTablet for tablet 3a551b13451448beb6f86f876946abc1 (DEFAULT_TABLE table=test-table [id=c5728d6b7e484fafb776f671287b390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.483788 29986 ts_tablet_manager.cc:1918] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:32.483902 29986 log.cc:1196] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/aa7591ffff5c4db0943333b4d592bcc3
I20260925 19:09:32.483896 29985 raft_consensus.cc:2229] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:32.483934 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a551b13451448beb6f86f876946abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.484009 29985 raft_consensus.cc:2258] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.484167 29986 ts_tablet_manager.cc:1939] T aa7591ffff5c4db0943333b4d592bcc3 P 2454883df6254294a5c187e36701594f: Deleting consensus metadata
I20260925 19:09:32.484762 29985 ts_tablet_manager.cc:1905] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.485188 29964 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Bootstrap starting.
I20260925 19:09:32.485266 29444 catalog_manager.cc:4841] TS 2454883df6254294a5c187e36701594f (127.27.174.67:43563): tablet aa7591ffff5c4db0943333b4d592bcc3 (table test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]) successfully deleted
I20260925 19:09:32.485430 29597 tablet_service.cc:1467] Processing CreateTablet for tablet 3a551b13451448beb6f86f876946abc1 (DEFAULT_TABLE table=test-table [id=c5728d6b7e484fafb776f671287b390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:32.485526 29597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a551b13451448beb6f86f876946abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.485569 29964 tablet_bootstrap.cc:654] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.486178 29985 ts_tablet_manager.cc:1918] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:32.486235 29985 log.cc:1196] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/aa7591ffff5c4db0943333b4d592bcc3
I20260925 19:09:32.486459 29985 ts_tablet_manager.cc:1939] T aa7591ffff5c4db0943333b4d592bcc3 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting consensus metadata
I20260925 19:09:32.486763 29963 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Bootstrap starting.
I20260925 19:09:32.487124 29963 tablet_bootstrap.cc:654] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.487162 29447 catalog_manager.cc:4841] TS 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731): tablet aa7591ffff5c4db0943333b4d592bcc3 (table test-table [id=a2c224ac84f0474b80c03d8ea0feb3c6]) successfully deleted
I20260925 19:09:32.487648 29965 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: No bootstrap required, opened a new log
I20260925 19:09:32.487691 29965 ts_tablet_manager.cc:1397] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 19:09:32.487709 29963 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: No bootstrap required, opened a new log
I20260925 19:09:32.487744 29964 tablet_bootstrap.cc:492] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: No bootstrap required, opened a new log
I20260925 19:09:32.487748 29963 ts_tablet_manager.cc:1397] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:32.487777 29964 ts_tablet_manager.cc:1397] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:09:32.487828 29965 raft_consensus.cc:348] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.487886 29965 raft_consensus.cc:374] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.487905 29965 raft_consensus.cc:729] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e354af5a54d3bb2f98528ff9f90fd, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.487897 29964 raft_consensus.cc:348] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.487897 29963 raft_consensus.cc:348] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.487968 29964 raft_consensus.cc:374] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.487968 29963 raft_consensus.cc:374] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.487962 29965 consensus_queue.cc:260] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.487990 29964 raft_consensus.cc:729] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2454883df6254294a5c187e36701594f, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.487990 29963 raft_consensus.cc:729] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d06dc02b0a4c24aaea193901cb501b, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.488035 29965 ts_tablet_manager.cc:1428] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.488031 29964 consensus_queue.cc:260] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.488030 29963 consensus_queue.cc:260] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.488102 29964 ts_tablet_manager.cc:1428] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.488102 29963 ts_tablet_manager.cc:1428] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 19:09:32.514691 29664 tablet.cc:2369] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:32.558625 29969 raft_consensus.cc:482] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:32.558712 29969 raft_consensus.cc:504] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.559053 29969 leader_election.cc:288] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), 2454883df6254294a5c187e36701594f (127.27.174.67:43563)
I20260925 19:09:32.562374 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.559260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48279 (local address 127.27.174.66:38731)
0925 19:09:32.559330 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.559332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.559414 (+ 82us) server_negotiation.cc:407] Connection header received
0925 19:09:32.559507 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.559508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.559516 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.559544 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.559810 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.560494 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.561661 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.562026 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.562108 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.562117 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.562266 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.562306 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:09:32.562386 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.559210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38731 (local address 127.27.174.65:48279)
0925 19:09:32.559351 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.559362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.559436 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.559630 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.559631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.559633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.559722 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.559725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.560585 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.560586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.561596 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.561598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.562118 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.562119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.562193 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.562196 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.562261 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.562308 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":17,"thread_start_us":73,"threads_started":1}
I20260925 19:09:32.562701 29973 raft_consensus.cc:482] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:32.562672 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd" is_pre_election: true
I20260925 19:09:32.562758 29757 raft_consensus.cc:2454] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 0.
I20260925 19:09:32.562759 29973 raft_consensus.cc:504] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.562904 29973 leader_election.cc:288] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591), 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731)
I20260925 19:09:32.562922 29551 leader_election.cc:302] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e354af5a54d3bb2f98528ff9f90fd, d1d06dc02b0a4c24aaea193901cb501b; no voters:
I20260925 19:09:32.563074 29969 raft_consensus.cc:2781] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:32.563115 29969 raft_consensus.cc:482] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:32.563093 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd" is_pre_election: true
I20260925 19:09:32.563134 29969 raft_consensus.cc:3037] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.563154 29757 raft_consensus.cc:2454] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2454883df6254294a5c187e36701594f in term 0.
I20260925 19:09:32.563161 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.559392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48381 (local address 127.27.174.67:43563)
0925 19:09:32.559433 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.559435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.559444 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:32.559497 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.559499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.559510 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.559543 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:32.559815 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.560260 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.561642 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.562227 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.562315 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.562324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.563046 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.563092 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:32.563285 29831 leader_election.cc:302] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2454883df6254294a5c187e36701594f, 430e354af5a54d3bb2f98528ff9f90fd; no voters:
I20260925 19:09:32.563408 29973 raft_consensus.cc:2781] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:32.563439 29973 raft_consensus.cc:482] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:32.563457 29973 raft_consensus.cc:3037] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.563490 29617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d06dc02b0a4c24aaea193901cb501b" is_pre_election: true
I20260925 19:09:32.563845 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.559245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:43563 (local address 127.27.174.65:48381)
0925 19:09:32.559372 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.559382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.559431 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.559630 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.559631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.559635 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:32.559722 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.559725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.560358 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.560359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.561550 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.561553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.562365 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.562367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.562835 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.562841 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.563704 (+ 863us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.563752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":57,"thread_start_us":53,"threads_started":1}
I20260925 19:09:32.564091 29969 raft_consensus.cc:504] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.564231 29969 leader_election.cc:288] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 election: Requested vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), 2454883df6254294a5c187e36701594f (127.27.174.67:43563)
I20260925 19:09:32.564213 29897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2454883df6254294a5c187e36701594f" is_pre_election: true
I20260925 19:09:32.564303 29973 raft_consensus.cc:504] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.564414 29897 raft_consensus.cc:2379] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1d06dc02b0a4c24aaea193901cb501b in current term 1: Already voted for candidate 2454883df6254294a5c187e36701594f in this term.
I20260925 19:09:32.564404 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd"
I20260925 19:09:32.564463 29757 raft_consensus.cc:3037] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.564560 29617 raft_consensus.cc:2379] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2454883df6254294a5c187e36701594f in current term 1: Already voted for candidate d1d06dc02b0a4c24aaea193901cb501b in this term.
I20260925 19:09:32.564590 29756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd"
I20260925 19:09:32.565218 29757 raft_consensus.cc:2454] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 1.
I20260925 19:09:32.565215 29617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "2454883df6254294a5c187e36701594f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d06dc02b0a4c24aaea193901cb501b"
I20260925 19:09:32.565289 29617 raft_consensus.cc:2379] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2454883df6254294a5c187e36701594f in current term 1: Already voted for candidate d1d06dc02b0a4c24aaea193901cb501b in this term.
I20260925 19:09:32.565281 29897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a551b13451448beb6f86f876946abc1" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2454883df6254294a5c187e36701594f"
I20260925 19:09:32.565351 29897 raft_consensus.cc:2379] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1d06dc02b0a4c24aaea193901cb501b in current term 1: Already voted for candidate 2454883df6254294a5c187e36701594f in this term.
I20260925 19:09:32.565364 29551 leader_election.cc:302] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e354af5a54d3bb2f98528ff9f90fd, d1d06dc02b0a4c24aaea193901cb501b; no voters:
I20260925 19:09:32.565433 29969 raft_consensus.cc:2781] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:32.565431 29829 leader_election.cc:302] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2454883df6254294a5c187e36701594f; no voters: 430e354af5a54d3bb2f98528ff9f90fd, d1d06dc02b0a4c24aaea193901cb501b
I20260925 19:09:32.564426 29973 leader_election.cc:288] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [CANDIDATE]: Term 1 election: Requested vote from peers d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591), 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731)
I20260925 19:09:32.565523 29969 raft_consensus.cc:686] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 LEADER]: Becoming Leader. State: Replica: d1d06dc02b0a4c24aaea193901cb501b, State: Running, Role: LEADER
I20260925 19:09:32.565501 29971 raft_consensus.cc:2726] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 19:09:32.565578 29969 consensus_queue.cc:237] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.566190 29459 catalog_manager.cc:5495] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b reported cstate change: term changed from 0 to 1, leader changed from <none> to d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65). New cstate: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } health_report { overall_health: UNKNOWN } } }
W20260925 19:09:32.581856 29944 tablet.cc:2369] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:32.607151 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.600296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38805 (local address 127.0.0.1:37322)
0925 19:09:32.600504 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.600514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.600616 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.600790 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.600791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.601143 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:32.601188 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.601190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.601864 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.601865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.602384 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.602385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.602729 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.602730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.602788 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.603362 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:32.603367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:32.604810 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:32.606118 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.606125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.606127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.606327 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.606496 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:32.606496 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:32.606497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:32.606550 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:32.606807 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:32.606813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:32.606855 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.606922 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.607038 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
I20260925 19:09:32.607159 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.600366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.27.174.126:38805)
0925 19:09:32.600454 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.600456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.600594 (+ 138us) server_negotiation.cc:407] Connection header received
0925 19:09:32.600696 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.600697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.600708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.600785 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.601289 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.601774 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.602469 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.602648 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.602720 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.604897 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.604898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.604899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.604905 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.606049 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.606392 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.606393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.606393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.606440 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.606602 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.606603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.606603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.606737 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.606788 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.606967 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.607083 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":35}
I20260925 19:09:32.609225 29459 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:37322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:09:32.609454 29459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:32.610888 29597 tablet_service.cc:1467] Processing CreateTablet for tablet 82033bb7f52d4827932679adfdd5a474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01802ec2e954f5c9761891e6a805764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:32.610900 29877 tablet_service.cc:1467] Processing CreateTablet for tablet 82033bb7f52d4827932679adfdd5a474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01802ec2e954f5c9761891e6a805764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:32.610913 29737 tablet_service.cc:1467] Processing CreateTablet for tablet 82033bb7f52d4827932679adfdd5a474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01802ec2e954f5c9761891e6a805764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:32.611042 29597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82033bb7f52d4827932679adfdd5a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.611047 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82033bb7f52d4827932679adfdd5a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.611063 29737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82033bb7f52d4827932679adfdd5a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:32.612289 29965 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: Bootstrap starting.
I20260925 19:09:32.612443 29963 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: Bootstrap starting.
I20260925 19:09:32.612671 29965 tablet_bootstrap.cc:654] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.612824 29963 tablet_bootstrap.cc:654] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.613178 29964 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: Bootstrap starting.
I20260925 19:09:32.613343 29965 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: No bootstrap required, opened a new log
I20260925 19:09:32.613399 29965 ts_tablet_manager.cc:1397] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:32.613482 29964 tablet_bootstrap.cc:654] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:32.613567 29965 raft_consensus.cc:348] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.613637 29965 raft_consensus.cc:374] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.613662 29965 raft_consensus.cc:729] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e354af5a54d3bb2f98528ff9f90fd, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.613719 29965 consensus_queue.cc:260] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.613816 29965 ts_tablet_manager.cc:1428] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.613880 29970 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.613965 29970 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.614003 29963 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: No bootstrap required, opened a new log
I20260925 19:09:32.614056 29963 ts_tablet_manager.cc:1397] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:09:32.614197 29963 raft_consensus.cc:348] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.614265 29963 raft_consensus.cc:374] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.614290 29963 raft_consensus.cc:729] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d06dc02b0a4c24aaea193901cb501b, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.614346 29963 consensus_queue.cc:260] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.614451 29963 ts_tablet_manager.cc:1428] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:32.614565 29969 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.614656 29969 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.615089 29964 tablet_bootstrap.cc:492] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: No bootstrap required, opened a new log
I20260925 19:09:32.615163 29964 ts_tablet_manager.cc:1397] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:32.615286 29964 raft_consensus.cc:348] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.615375 29964 raft_consensus.cc:374] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:32.615408 29964 raft_consensus.cc:729] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2454883df6254294a5c187e36701594f, State: Initialized, Role: FOLLOWER
I20260925 19:09:32.615458 29964 consensus_queue.cc:260] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.615545 29964 ts_tablet_manager.cc:1428] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:32.615578 29971 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.615692 29971 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.837047 29969 raft_consensus.cc:482] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:32.837131 29969 raft_consensus.cc:504] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.837272 29969 leader_election.cc:288] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), 2454883df6254294a5c187e36701594f (127.27.174.67:43563)
I20260925 19:09:32.837487 29897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82033bb7f52d4827932679adfdd5a474" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2454883df6254294a5c187e36701594f" is_pre_election: true
I20260925 19:09:32.837517 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82033bb7f52d4827932679adfdd5a474" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd" is_pre_election: true
I20260925 19:09:32.837600 29757 raft_consensus.cc:2454] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 0.
I20260925 19:09:32.837607 29897 raft_consensus.cc:2454] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 0.
I20260925 19:09:32.837739 29551 leader_election.cc:302] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e354af5a54d3bb2f98528ff9f90fd, d1d06dc02b0a4c24aaea193901cb501b; no voters:
I20260925 19:09:32.837854 29969 raft_consensus.cc:2781] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:32.837896 29969 raft_consensus.cc:482] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:32.837924 29969 raft_consensus.cc:3037] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.838619 29969 raft_consensus.cc:504] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.838786 29969 leader_election.cc:288] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 election: Requested vote from peers 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731), 2454883df6254294a5c187e36701594f (127.27.174.67:43563)
I20260925 19:09:32.838904 29757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82033bb7f52d4827932679adfdd5a474" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e354af5a54d3bb2f98528ff9f90fd"
I20260925 19:09:32.838922 29897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82033bb7f52d4827932679adfdd5a474" candidate_uuid: "d1d06dc02b0a4c24aaea193901cb501b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2454883df6254294a5c187e36701594f"
I20260925 19:09:32.838975 29757 raft_consensus.cc:3037] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.838981 29897 raft_consensus.cc:3037] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:32.839743 29757 raft_consensus.cc:2454] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 1.
I20260925 19:09:32.839856 29897 raft_consensus.cc:2454] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d06dc02b0a4c24aaea193901cb501b in term 1.
I20260925 19:09:32.839910 29551 leader_election.cc:302] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e354af5a54d3bb2f98528ff9f90fd, d1d06dc02b0a4c24aaea193901cb501b; no voters:
I20260925 19:09:32.840004 29969 raft_consensus.cc:2781] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:32.840049 29969 raft_consensus.cc:686] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [term 1 LEADER]: Becoming Leader. State: Replica: d1d06dc02b0a4c24aaea193901cb501b, State: Running, Role: LEADER
I20260925 19:09:32.840108 29969 consensus_queue.cc:237] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } }
I20260925 19:09:32.840427 29993 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: TxnStatusTablet state changed. Reason: New leader d1d06dc02b0a4c24aaea193901cb501b. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.840535 29993 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:32.840757 30005 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:09:32.840911 30005 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:09:32.841022 29460 catalog_manager.cc:5495] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b reported cstate change: term changed from 0 to 1, leader changed from <none> to d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65). New cstate: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:32.899740 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.896994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41591 (local address 127.0.0.1:52846)
0925 19:09:32.897061 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.897074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.897144 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.897448 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.897449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.897450 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:32.897502 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.897504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.898159 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.898160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.898756 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.898758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.899151 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.899152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.899219 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.899234 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.899532 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.899542 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.899591 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.899672 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:09:32.899722 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.897239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.27.174.65:41591)
0925 19:09:32.897297 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.897299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.897306 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:32.897339 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.897340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.897365 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.897441 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:32.897590 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.898074 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.898857 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.899061 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.899149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.899337 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.899442 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.899545 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.899618 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.899643 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:32.900944 29757 raft_consensus.cc:1261] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Refusing update from remote peer d1d06dc02b0a4c24aaea193901cb501b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.900971 29897 raft_consensus.cc:1261] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Refusing update from remote peer d1d06dc02b0a4c24aaea193901cb501b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.901180 29993 consensus_queue.cc:1035] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Connected to new peer: Peer: permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.901338 29993 consensus_queue.cc:1035] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.901865 29970 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: TxnStatusTablet state changed. Reason: New leader d1d06dc02b0a4c24aaea193901cb501b. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.901953 29970 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.901928 29971 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: TxnStatusTablet state changed. Reason: New leader d1d06dc02b0a4c24aaea193901cb501b. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.902015 29971 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.902683 29993 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.902761 29993 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:32.902875 29971 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.902966 29971 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 2454883df6254294a5c187e36701594f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.903838 29969 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.903921 29969 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P d1d06dc02b0a4c24aaea193901cb501b: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:32.904258 29970 tablet_replica.cc:439] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1d06dc02b0a4c24aaea193901cb501b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d06dc02b0a4c24aaea193901cb501b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41591 } } peers { permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 } } peers { permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 } } }
I20260925 19:09:32.904353 29970 tablet_replica.cc:442] T 82033bb7f52d4827932679adfdd5a474 P 430e354af5a54d3bb2f98528ff9f90fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:32.906966 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.904719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.27.174.65:41591)
0925 19:09:32.904753 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.904755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.904764 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:32.904784 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.904784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.904794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.904838 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:32.905010 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.905468 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.906250 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.906455 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.906531 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.906680 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.906733 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.906792 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.906875 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.906892 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:32.911728 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.909412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35505 (local address 127.27.174.65:41591)
0925 19:09:32.909483 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.909485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.909497 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:32.909569 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.909570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.909578 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.909639 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:32.909821 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.910268 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.911028 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.911219 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.911299 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.911462 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.911489 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.911550 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.911628 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.911652 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260925 19:09:32.911952 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.909287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41591 (local address 127.27.174.65:35505)
0925 19:09:32.909414 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:32.909423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:32.909511 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:32.909633 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:32.909633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:32.909636 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:32.909752 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.909754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.910356 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.910357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.910934 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:32.910935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.911301 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.911301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:32.911364 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.911375 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:32.911563 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:32.911564 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:32.911626 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:09:32.911695 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":51,"thread_start_us":72,"threads_started":1}
I20260925 19:09:32.914170 29897 raft_consensus.cc:1261] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Refusing update from remote peer d1d06dc02b0a4c24aaea193901cb501b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.914191 29757 raft_consensus.cc:1261] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Refusing update from remote peer d1d06dc02b0a4c24aaea193901cb501b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:32.914359 29993 consensus_queue.cc:1035] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2454883df6254294a5c187e36701594f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.914443 29969 consensus_queue.cc:1035] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [LEADER]: Connected to new peer: Peer: permanent_uuid: "430e354af5a54d3bb2f98528ff9f90fd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:32.926680 29459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:32.926805 29459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:32.927450 29459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6208e4c743ad474baaba96b15a936121: Sending DeleteTablet for 3 replicas of tablet 3a551b13451448beb6f86f876946abc1
I20260925 19:09:32.927757 29877 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a551b13451448beb6f86f876946abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56722
I20260925 19:09:32.927861 29986 tablet_replica.cc:330] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: stopping tablet replica
I20260925 19:09:32.927917 29986 raft_consensus.cc:2229] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:32.927955 29986 raft_consensus.cc:2258] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.928220 29986 ts_tablet_manager.cc:1905] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.928390 29737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a551b13451448beb6f86f876946abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33556
I20260925 19:09:32.928429 29597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a551b13451448beb6f86f876946abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:32 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52834
I20260925 19:09:32.928508 29987 tablet_replica.cc:330] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: stopping tablet replica
I20260925 19:09:32.928566 29987 raft_consensus.cc:2229] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:32.928648 29987 raft_consensus.cc:2258] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.929790 29985 tablet_replica.cc:330] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: stopping tablet replica
I20260925 19:09:32.929862 29985 raft_consensus.cc:2229] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:32.929898 29985 raft_consensus.cc:2258] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:32.930073 29986 ts_tablet_manager.cc:1918] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:32.930133 29986 log.cc:1196] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/3a551b13451448beb6f86f876946abc1
I20260925 19:09:32.930291 29987 ts_tablet_manager.cc:1905] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.930371 29986 ts_tablet_manager.cc:1939] T 3a551b13451448beb6f86f876946abc1 P 2454883df6254294a5c187e36701594f: Deleting consensus metadata
I20260925 19:09:32.930737 29444 catalog_manager.cc:4841] TS 2454883df6254294a5c187e36701594f (127.27.174.67:43563): tablet 3a551b13451448beb6f86f876946abc1 (table test-table [id=c5728d6b7e484fafb776f671287b390c]) successfully deleted
I20260925 19:09:32.931694 29985 ts_tablet_manager.cc:1905] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:32.931694 29987 ts_tablet_manager.cc:1918] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:32.931746 29987 log.cc:1196] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/3a551b13451448beb6f86f876946abc1
I20260925 19:09:32.931970 29987 ts_tablet_manager.cc:1939] T 3a551b13451448beb6f86f876946abc1 P d1d06dc02b0a4c24aaea193901cb501b: Deleting consensus metadata
I20260925 19:09:32.932281 29445 catalog_manager.cc:4841] TS d1d06dc02b0a4c24aaea193901cb501b (127.27.174.65:41591): tablet 3a551b13451448beb6f86f876946abc1 (table test-table [id=c5728d6b7e484fafb776f671287b390c]) successfully deleted
I20260925 19:09:32.933281 29985 ts_tablet_manager.cc:1918] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:32.933372 29985 log.cc:1196] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/3a551b13451448beb6f86f876946abc1
I20260925 19:09:32.933579 29985 ts_tablet_manager.cc:1939] T 3a551b13451448beb6f86f876946abc1 P 430e354af5a54d3bb2f98528ff9f90fd: Deleting consensus metadata
I20260925 19:09:32.933878 29447 catalog_manager.cc:4841] TS 430e354af5a54d3bb2f98528ff9f90fd (127.27.174.66:38731): tablet 3a551b13451448beb6f86f876946abc1 (table test-table [id=c5728d6b7e484fafb776f671287b390c]) successfully deleted
Sep 25 19:09:32 dist-test-slave-fcdn krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:32.944271 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.930219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.27.174.65:41591)
0925 19:09:32.933064 (+ 2845us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.933066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.935818 (+ 2752us) server_negotiation.cc:407] Connection header received
0925 19:09:32.935846 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.935847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.935859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.935919 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.936432 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.937161 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.937852 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.938140 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.938212 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.941226 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.941233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.941233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.941241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.943040 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.943375 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.943379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.943380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.943455 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.943702 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.943703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.943704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.943870 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.944036 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.944084 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.944163 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2824}
W20260925 19:09:32.944526 29657 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52866
I20260925 19:09:32.951905 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:32.945529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.27.174.126:38805)
0925 19:09:32.945574 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:32.945576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:32.945711 (+ 135us) server_negotiation.cc:407] Connection header received
0925 19:09:32.945763 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:32.945764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:32.945772 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:32.945845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:32.946449 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.947033 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.948011 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:32.948181 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:32.948231 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:32.949451 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:32.949452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:32.949453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:32.949458 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:32.950829 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.951205 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.951206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.951206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.951253 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:32.951449 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:32.951450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:32.951450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:32.951541 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:32.951676 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:32.951781 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:32.951845 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:09:32.952104 29459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37326
I20260925 19:09:32.952734 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29532
I20260925 19:09:32.958772 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29672
I20260925 19:09:32.963241 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29812
I20260925 19:09:32.968571 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 29428
2026-09-25T19:09:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2404 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:33 dist-test-slave-fcdn krb5kdc[30024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:33 dist-test-slave-fcdn krb5kdc[30024](info): set up 2 sockets
Sep 25 19:09:33 dist-test-slave-fcdn krb5kdc[30024](info): commencing operation
krb5kdc: starting...
W20260925 19:09:34.003212 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:34.122092 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40645
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:34.202730 30037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:34.202978 30037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:34.203017 30037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:34.204260 30037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:34.204322 30037 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:09:34.204346 30037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:34.204378 30037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:34.204413 30037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:34.204440 30037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:34.205927 30037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:40645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:34.206208 30037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:34.206434 30037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:34.211330 30043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.211318 30045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.211318 30042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:34.211345 30037 server_base.cc:1034] running on GCE node
I20260925 19:09:34.211808 30037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:34.212121 30037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:34.213269 30037 hybrid_clock.cc:648] HybridClock initialized: now 1790363374213253 us; error 35 us; skew 500 ppm
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:34.215742 30037 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:34.216203 30037 webserver.cc:460] Webserver started at http://127.27.174.126:36301/ using document root <none> and password file <none>
I20260925 19:09:34.216447 30037 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:34.216514 30037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:34.216648 30037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:34.217691 30037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "02818e07fedc4b96ae0be37b93c6710c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "a36e7680429e30f8d32077afa07578af"
server_key_iv: "b96f14350874ea2a83a726112ceef1fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.218063 30037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "02818e07fedc4b96ae0be37b93c6710c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "a36e7680429e30f8d32077afa07578af"
server_key_iv: "b96f14350874ea2a83a726112ceef1fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.219388 30037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:34.220239 30052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.220418 30037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:09:34.220515 30037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "02818e07fedc4b96ae0be37b93c6710c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "a36e7680429e30f8d32077afa07578af"
server_key_iv: "b96f14350874ea2a83a726112ceef1fe"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.220602 30037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:34.252748 30037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:34.255111 30037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:34.255307 30037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:34.260121 30037 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:40645
I20260925 19:09:34.260205 30114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:40645 every 8 connection(s)
I20260925 19:09:34.260505 30037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:34.261160 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.263362 30115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Bootstrap starting.
I20260925 19:09:34.263993 30115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.264242 30115 log.cc:824] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:34.264961 30115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: No bootstrap required, opened a new log
I20260925 19:09:34.267160 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30037
I20260925 19:09:34.267266 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:34.267436 28345 external_mini_cluster.cc:1428] Setting key 89445caa68b41ad2f90a5d858a5f5285
I20260925 19:09:34.267411 30115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } }
I20260925 19:09:34.267652 30115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.267689 30115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02818e07fedc4b96ae0be37b93c6710c, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.267841 30115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } }
I20260925 19:09:34.267923 30115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:34.267956 30115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:34.268028 30115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:34.268781 30115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } }
I20260925 19:09:34.268922 30115 leader_election.cc:302] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02818e07fedc4b96ae0be37b93c6710c; no voters:
I20260925 19:09:34.269100 30115 leader_election.cc:288] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:34.269189 30120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:34.269380 30115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:34.269387 30120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [term 1 LEADER]: Becoming Leader. State: Replica: 02818e07fedc4b96ae0be37b93c6710c, State: Running, Role: LEADER
I20260925 19:09:34.269626 30120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } }
I20260925 19:09:34.270051 30122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02818e07fedc4b96ae0be37b93c6710c. Latest consensus state: current_term: 1 leader_uuid: "02818e07fedc4b96ae0be37b93c6710c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } } }
I20260925 19:09:34.270125 30122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:34.270555 30121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02818e07fedc4b96ae0be37b93c6710c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02818e07fedc4b96ae0be37b93c6710c" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40645 } } }
I20260925 19:09:34.270622 30121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:34.271214 30135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:34.271281 30135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:34.271374 30129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:34.271546 30129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:34.273351 30129 catalog_manager.cc:1347] Generated new cluster ID: 800bea4319d24c1ebb91da149b880e11
I20260925 19:09:34.273424 30129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.279407 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.267979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.27.174.126:40645)
0925 19:09:34.268172 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.268182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.268217 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:09:34.268341 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.268347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.268399 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.268513 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.269322 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.270945 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.272057 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.272601 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.272727 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.275641 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.275646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.275646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.275665 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.277725 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.278159 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.278162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.278163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.278279 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.278493 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.278493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.278494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.278803 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.278945 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.279074 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.279206 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260925 19:09:34.283687 30129 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:34.284412 30129 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:34.293342 30129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Generated new TSK 0
I20260925 19:09:34.293596 30129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:34.326615 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40645
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:34.424391 30141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:34.424692 30141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:34.424739 30141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:34.425951 30141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:34.426023 30141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:34.426061 30141 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:09:34.426091 30141 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:34.426124 30141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:34.427891 30141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:34.428190 30141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:34.428432 30141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:34.433635 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.433710 30146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:34.433851 30141 server_base.cc:1034] running on GCE node
W20260925 19:09:34.433616 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:34.434134 30141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:34.434396 30141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:34.435544 30141 hybrid_clock.cc:648] HybridClock initialized: now 1790363374435531 us; error 37 us; skew 500 ppm
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:34.438136 30141 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:34.438607 30141 webserver.cc:460] Webserver started at http://127.27.174.65:46117/ using document root <none> and password file <none>
I20260925 19:09:34.438838 30141 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:34.438882 30141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:34.439051 30141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:34.440197 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "31204205fc034c039266928d1216782e"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "fdf7b081af1a770decbf4fccfa0e8684"
server_key_iv: "bb38cf3ab33ba0b619afb58097555d0a"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.440567 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "31204205fc034c039266928d1216782e"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "fdf7b081af1a770decbf4fccfa0e8684"
server_key_iv: "bb38cf3ab33ba0b619afb58097555d0a"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.441874 30141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:34.442637 30156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.442847 30141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:34.442951 30141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "31204205fc034c039266928d1216782e"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "fdf7b081af1a770decbf4fccfa0e8684"
server_key_iv: "bb38cf3ab33ba0b619afb58097555d0a"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.443024 30141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:34.456017 30141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:34.458340 30141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:34.458505 30141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:34.459025 30141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:34.459095 30141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.459139 30141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:34.459169 30141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.466048 30141 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44983
I20260925 19:09:34.466117 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44983 every 8 connection(s)
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.466428 30141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:34.469444 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.459570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33431 (local address 127.27.174.126:40645)
0925 19:09:34.459649 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.459651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.459823 (+ 172us) server_negotiation.cc:407] Connection header received
0925 19:09:34.460331 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.460332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.460346 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.460427 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.461261 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.462110 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.463713 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.464076 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.464159 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.466673 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.466674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.466675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.466691 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.468025 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.468436 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.468437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.468437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.468502 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.468755 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.468756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.468756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.468879 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.468957 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.469266 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.469355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:09:34.469610 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.459525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.65:33431)
0925 19:09:34.459686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.459728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.460236 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.460491 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.460495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.460851 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.461134 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.461141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.462216 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.462217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.463626 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.463630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.464172 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.464173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.464323 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.464863 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.464874 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.466572 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.468112 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.468118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.468120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.468346 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.468567 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.468568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.468568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.468682 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.468947 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.468953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.469091 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.469227 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.469383 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":106,"threads_started":1}
I20260925 19:09:34.470173 30272 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40645
I20260925 19:09:34.470257 30272 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:34.470444 30272 heartbeater.cc:511] Master 127.27.174.126:40645 requested a full tablet report, sending...
I20260925 19:09:34.471009 30068 ts_manager.cc:194] Registered new tserver with Master: 31204205fc034c039266928d1216782e (127.27.174.65:44983)
I20260925 19:09:34.471815 30068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:33431
I20260925 19:09:34.471931 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30141
I20260925 19:09:34.472024 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:34.472224 28345 external_mini_cluster.cc:1428] Setting key d7dd9aab85305d27c69565e6d024acae
I20260925 19:09:34.477768 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.471294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49123 (local address 127.27.174.126:40645)
0925 19:09:34.471667 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.471669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.471678 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:34.471698 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.471698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.471707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.471786 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.472175 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.472842 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.473846 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.474028 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.474088 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.475038 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.475039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.475040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.475047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.476652 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.477048 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.477049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.477049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.477111 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.477298 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.477299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.477300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.477441 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.477510 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.477649 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.477708 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":354}
I20260925 19:09:34.477873 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.471225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.65:49123)
0925 19:09:34.471335 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.471345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.471397 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.471799 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.471800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.472006 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.472074 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.472076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.472935 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.472935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.473765 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.473767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.474109 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.474110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.474192 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.474677 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.474681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.474945 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.476736 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.476736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.476737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.476970 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.477179 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.477184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.477185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.477237 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.477511 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.477512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.477559 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.477632 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.477744 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:34.498692 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40645
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:34.579195 30281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:34.579406 30281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:34.579432 30281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:34.580641 30281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:34.580683 30281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:34.580704 30281 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:09:34.580716 30281 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:34.580734 30281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:34.582182 30281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:34.582427 30281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:34.582665 30281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:34.587917 30286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.587960 30287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.587934 30289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:34.588235 30281 server_base.cc:1034] running on GCE node
I20260925 19:09:34.588423 30281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:34.588613 30281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:34.589790 30281 hybrid_clock.cc:648] HybridClock initialized: now 1790363374589775 us; error 27 us; skew 500 ppm
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:34.592240 30281 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:34.592657 30281 webserver.cc:460] Webserver started at http://127.27.174.66:38169/ using document root <none> and password file <none>
I20260925 19:09:34.592903 30281 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:34.592975 30281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:34.593128 30281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:34.594178 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c5180e4badca40c69f2fbafd4d75828c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "7c822bda96fba8678d7225483e03468d"
server_key_iv: "e7e6b8d80fb661074747f0190abf3e60"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.594534 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c5180e4badca40c69f2fbafd4d75828c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "7c822bda96fba8678d7225483e03468d"
server_key_iv: "e7e6b8d80fb661074747f0190abf3e60"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.595911 30281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:09:34.596621 30296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.596791 30281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.596895 30281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "c5180e4badca40c69f2fbafd4d75828c"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "7c822bda96fba8678d7225483e03468d"
server_key_iv: "e7e6b8d80fb661074747f0190abf3e60"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.596971 30281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:34.618396 30281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:34.620852 30281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:34.621021 30281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:34.621531 30281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:34.621598 30281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.621644 30281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:34.621667 30281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.629036 30281 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44823
I20260925 19:09:34.629105 30411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44823 every 8 connection(s)
I20260925 19:09:34.629446 30281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:34.632052 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.622116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:59019 (local address 127.27.174.126:40645)
0925 19:09:34.622180 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.622182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.622380 (+ 198us) server_negotiation.cc:407] Connection header received
0925 19:09:34.622935 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.622937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.622951 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.623023 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.624006 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.624829 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.626491 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.626714 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.626773 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.629378 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.629379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.629380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.629388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.630830 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.631233 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.631234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.631234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.631291 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.631498 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.631498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.631499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.631650 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.631716 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.631933 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.631991 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:09:34.632313 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.622079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.66:59019)
0925 19:09:34.622246 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.622290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.622813 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.623123 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.623128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.623529 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.623891 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.623899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.624927 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.624928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.626370 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.626373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.626794 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.626795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.626956 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.627495 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.627506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.629283 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.630917 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.630923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.630924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.631151 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.631346 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.631347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.631347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.631442 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.631702 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.631707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.631828 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.631917 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.632040 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260925 19:09:34.632812 30412 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40645
I20260925 19:09:34.632893 30412 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:34.633070 30412 heartbeater.cc:511] Master 127.27.174.126:40645 requested a full tablet report, sending...
I20260925 19:09:34.633411 30069 ts_manager.cc:194] Registered new tserver with Master: c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823)
I20260925 19:09:34.633778 30069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:59019
I20260925 19:09:34.633893 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30281
I20260925 19:09:34.633982 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:34.634191 28345 external_mini_cluster.cc:1428] Setting key 56a801f0bcd1824da7580f6214296ca7
I20260925 19:09:34.639556 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.633826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.66:38635)
0925 19:09:34.634054 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.634064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.634117 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.634233 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.634234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.634447 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.634510 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.634512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.635231 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.635232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.635804 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.635806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.636169 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.636170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.636251 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.636670 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.636674 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.636925 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.638583 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.638584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.638584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.638745 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.638953 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.638953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.638954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.639006 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.639340 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.639340 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.639374 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.639423 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.639495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":68,"threads_started":1}
I20260925 19:09:34.639684 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.633900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38635 (local address 127.27.174.126:40645)
0925 19:09:34.633993 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.633995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.634104 (+ 109us) server_negotiation.cc:407] Connection header received
0925 19:09:34.634173 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.634174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.634183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.634231 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.634611 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.635146 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.635884 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.636086 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.636167 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.637023 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.637025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.637025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.637032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.638505 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.638822 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.638823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.638823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.638883 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.639073 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.639074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.639074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.639269 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.639349 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.639452 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.639534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:34.660640 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40645
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:34.740283 30421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:34.740487 30421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:34.740510 30421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:34.741655 30421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:34.741696 30421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:34.741717 30421 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:09:34.741761 30421 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:34.741796 30421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:34.743198 30421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:34.743438 30421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:34.743691 30421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:34.748555 30429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.748739 30427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:34.748667 30426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:34.748895 30421 server_base.cc:1034] running on GCE node
I20260925 19:09:34.749076 30421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:34.749307 30421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:34.750466 30421 hybrid_clock.cc:648] HybridClock initialized: now 1790363374750454 us; error 23 us; skew 500 ppm
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:34.752825 30421 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:34.753281 30421 webserver.cc:460] Webserver started at http://127.27.174.67:41623/ using document root <none> and password file <none>
I20260925 19:09:34.753510 30421 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:34.753577 30421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:34.753726 30421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:34.754757 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "43a3345eeef4431f9b6229b1d75f26c2"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "8a81a4471ae471b52575023da8dfd3be"
server_key_iv: "26c5d2a35da83ccf87f5d61e41e872ce"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.755110 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "43a3345eeef4431f9b6229b1d75f26c2"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "8a81a4471ae471b52575023da8dfd3be"
server_key_iv: "26c5d2a35da83ccf87f5d61e41e872ce"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.756477 30421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:34.757165 30436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.757334 30421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:34.757436 30421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "43a3345eeef4431f9b6229b1d75f26c2"
format_stamp: "Formatted at 2026-09-25 19:09:34 on dist-test-slave-fcdn"
server_key: "8a81a4471ae471b52575023da8dfd3be"
server_key_iv: "26c5d2a35da83ccf87f5d61e41e872ce"
server_key_version: "encryptionkey@0"
I20260925 19:09:34.757505 30421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:34.772523 30421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:34.774485 30421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:34.774654 30421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:34.775197 30421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:34.775265 30421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.775306 30421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:34.775332 30421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.783092 30421 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46825
I20260925 19:09:34.783200 30551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46825 every 8 connection(s)
I20260925 19:09:34.783488 30421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:34.784593 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.775657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:51585 (local address 127.27.174.126:40645)
0925 19:09:34.775699 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.775701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.775908 (+ 207us) server_negotiation.cc:407] Connection header received
0925 19:09:34.776429 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.776430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.776445 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.776495 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.777073 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.777570 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.778564 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.778770 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.778859 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.781175 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.781178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.781178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.781187 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.783089 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.783640 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.783641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.783642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.783706 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.783963 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.783964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.783965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.784086 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.784192 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.784468 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.784532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:34.784746 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.775653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.67:51585)
0925 19:09:34.775792 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.775825 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.776315 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.776564 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.776567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.776806 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.776977 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.776981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.777668 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.777669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.778462 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.778464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.778868 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.778869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.778981 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.779480 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.779488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.781038 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.783196 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.783200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.783202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.783532 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.783782 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.783783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.783783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.783910 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.784190 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.784194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.784337 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.784424 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.784538 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260925 19:09:34.785287 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30421
I20260925 19:09:34.785391 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:34.785565 28345 external_mini_cluster.cc:1428] Setting key a0ab8e6d30ce5b9f0f5f281782f5f994
I20260925 19:09:34.785790 30552 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40645
I20260925 19:09:34.785874 30552 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:34.786067 30552 heartbeater.cc:511] Master 127.27.174.126:40645 requested a full tablet report, sending...
I20260925 19:09:34.786527 30068 ts_manager.cc:194] Registered new tserver with Master: 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825)
I20260925 19:09:34.786906 30068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:51585
I20260925 19:09:34.791702 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.786443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:45009 (local address 127.27.174.126:40645)
0925 19:09:34.786480 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.786482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.786603 (+ 121us) server_negotiation.cc:407] Connection header received
0925 19:09:34.786703 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.786704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.786713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.786780 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.787205 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.787745 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.788440 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.788656 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.788738 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.789518 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.789519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.789519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.789526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.790600 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.790939 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.790940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.790940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.790993 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.791196 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.791197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.791197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.791304 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.791379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.791529 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.791619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:09:34.791719 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.786401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.27.174.67:45009)
0925 19:09:34.786528 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.786538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.786617 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.786785 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.786786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.787049 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.787136 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.787138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.787839 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.787840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.788355 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.788356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.788739 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.788740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.788814 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.789171 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.789174 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.789429 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.790696 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.790697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.790697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.790866 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.791068 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.791069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.791069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.791127 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.791371 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.791371 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.791416 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.791506 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.791600 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 19:09:34.797116 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.819146 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.812049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.27.174.126:40645)
0925 19:09:34.812106 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.812109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.812119 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:34.812211 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.812212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.812229 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.812295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.812858 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.813334 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.814083 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.814289 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.814393 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.816473 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.816474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.816474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.816481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.818174 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.818580 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.818581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.818581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.818631 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.818789 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.818790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.818790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.818886 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.818957 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.819031 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.819085 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:34.820801 30068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:34.822342 30068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:34.828950 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.824684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46825 (local address 127.0.0.1:54760)
0925 19:09:34.825097 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.825107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.825150 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.825625 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.825625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.825627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:34.825672 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.825673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826792 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.826793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.827710 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.827713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828578 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.828683 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.828697 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.828794 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.828860 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260925 19:09:34.828963 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.824941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.27.174.67:46825)
0925 19:09:34.825424 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.825430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.825440 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:34.825476 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.825478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.825492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.825562 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:34.825742 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826708 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.827824 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828475 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828548 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.828618 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.828816 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.828858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":76,"threads_started":1}
I20260925 19:09:34.829210 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.824900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44983 (local address 127.0.0.1:52944)
0925 19:09:34.825133 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.825145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.825197 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.825634 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.825635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.825637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:34.825697 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.825699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826775 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.826776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.828053 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.828055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828948 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.829017 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.829021 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.829107 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.829154 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":27,"thread_start_us":74,"threads_started":1}
I20260925 19:09:34.829267 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.825106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.27.174.65:44983)
0925 19:09:34.825280 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.825285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.825294 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:34.825335 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.825336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.825349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.825512 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:34.825850 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826678 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828152 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828793 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828858 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.828913 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.829129 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.829176 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260925 19:09:34.829849 30206 tablet_service.cc:1467] Processing CreateTablet for tablet 08da33698184473782504656af440a9a (DEFAULT_TABLE table=test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:34.829828 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 08da33698184473782504656af440a9a (DEFAULT_TABLE table=test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:34.830139 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08da33698184473782504656af440a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.830160 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08da33698184473782504656af440a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.832017 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.824682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44823 (local address 127.0.0.1:52612)
0925 19:09:34.824820 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.824862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.824927 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.825214 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.825217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.825219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:34.825282 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.825283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826578 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.826579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.828035 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.828038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828941 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.831842 (+ 2900us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.831848 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.831894 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.831942 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260925 19:09:34.832469 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.824773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.27.174.66:44823)
0925 19:09:34.824908 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.824913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.824926 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:09:34.825030 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.825032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.825048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.825139 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:34.825367 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.826479 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.828126 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.828842 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.832214 (+ 3372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.832285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.832313 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.832355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260925 19:09:34.833050 30346 tablet_service.cc:1467] Processing CreateTablet for tablet 08da33698184473782504656af440a9a (DEFAULT_TABLE table=test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:34.833330 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08da33698184473782504656af440a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.833520 30575 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Bootstrap starting.
I20260925 19:09:34.834203 30575 tablet_bootstrap.cc:654] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.834527 30575 log.cc:824] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:34.835278 30575 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: No bootstrap required, opened a new log
I20260925 19:09:34.835335 30575 ts_tablet_manager.cc:1397] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:34.835887 30577 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Bootstrap starting.
I20260925 19:09:34.836602 30578 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Bootstrap starting.
I20260925 19:09:34.836953 30577 tablet_bootstrap.cc:654] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.837244 30577 log.cc:824] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:34.837255 30578 tablet_bootstrap.cc:654] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.837630 30578 log.cc:824] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:34.837805 30575 raft_consensus.cc:348] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.837997 30575 raft_consensus.cc:374] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.838045 30575 raft_consensus.cc:729] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31204205fc034c039266928d1216782e, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.838064 30577 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: No bootstrap required, opened a new log
I20260925 19:09:34.838124 30577 ts_tablet_manager.cc:1397] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:34.838176 30575 consensus_queue.cc:260] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.838335 30578 tablet_bootstrap.cc:492] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: No bootstrap required, opened a new log
I20260925 19:09:34.838397 30578 ts_tablet_manager.cc:1397] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:34.838467 30575 ts_tablet_manager.cc:1428] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:34.838547 30272 heartbeater.cc:503] Master 127.27.174.126:40645 was elected leader, sending a full tablet report...
I20260925 19:09:34.840339 30578 raft_consensus.cc:348] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.840390 30577 raft_consensus.cc:348] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.840497 30578 raft_consensus.cc:374] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.840508 30577 raft_consensus.cc:374] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.840538 30578 raft_consensus.cc:729] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5180e4badca40c69f2fbafd4d75828c, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.840539 30577 raft_consensus.cc:729] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.840648 30578 consensus_queue.cc:260] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.840662 30577 consensus_queue.cc:260] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.840904 30578 ts_tablet_manager.cc:1428] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:09:34.840904 30577 ts_tablet_manager.cc:1428] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:09:34.840953 30552 heartbeater.cc:503] Master 127.27.174.126:40645 was elected leader, sending a full tablet report...
I20260925 19:09:34.841029 30412 heartbeater.cc:503] Master 127.27.174.126:40645 was elected leader, sending a full tablet report...
W20260925 19:09:34.880540 30413 tablet.cc:2369] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:34.930523 30583 raft_consensus.cc:482] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:34.930655 30583 raft_consensus.cc:504] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.931064 30583 leader_election.cc:288] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 31204205fc034c039266928d1216782e (127.27.174.65:44983)
I20260925 19:09:34.934144 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.931374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52097 (local address 127.27.174.66:44823)
0925 19:09:34.931420 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.931422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.931587 (+ 165us) server_negotiation.cc:407] Connection header received
0925 19:09:34.931636 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.931637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.931646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.931704 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:34.931909 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.932469 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933436 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.933797 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933880 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.933889 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.934032 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.934067 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:34.934152 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.931299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44983 (local address 127.27.174.67:40393)
0925 19:09:34.931367 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.931379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.931427 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.931621 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.931622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.931624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:34.931698 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.931700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.932372 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.932373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.933337 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.933339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.933878 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.933956 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.933961 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.934036 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.934075 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":40}
I20260925 19:09:34.934158 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.931399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40393 (local address 127.27.174.65:44983)
0925 19:09:34.931442 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.931444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.931453 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:34.931483 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.931484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.931495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.931579 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:34.931774 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.932301 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933425 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.933793 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.933888 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.934056 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.934099 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:09:34.934165 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.931300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44823 (local address 127.27.174.67:52097)
0925 19:09:34.931500 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.931510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.931570 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.931724 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.931724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.931760 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:34.931834 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.931836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.932551 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.932552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.933339 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.933342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.933882 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.933883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.933958 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.933961 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.934004 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.934063 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":35,"thread_start_us":109,"threads_started":1}
I20260925 19:09:34.934523 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08da33698184473782504656af440a9a" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e" is_pre_election: true
I20260925 19:09:34.934722 30226 raft_consensus.cc:2454] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:34.934760 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08da33698184473782504656af440a9a" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c" is_pre_election: true
I20260925 19:09:34.934926 30366 raft_consensus.cc:2454] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:34.934975 30437 leader_election.cc:302] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, 43a3345eeef4431f9b6229b1d75f26c2; no voters:
I20260925 19:09:34.935086 30583 raft_consensus.cc:2781] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:34.935138 30583 raft_consensus.cc:482] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:34.935164 30583 raft_consensus.cc:3037] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:34.935869 30583 raft_consensus.cc:504] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.936007 30583 leader_election.cc:288] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Requested vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 31204205fc034c039266928d1216782e (127.27.174.65:44983)
I20260925 19:09:34.936173 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08da33698184473782504656af440a9a" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c"
I20260925 19:09:34.936178 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08da33698184473782504656af440a9a" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e"
I20260925 19:09:34.936249 30366 raft_consensus.cc:3037] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:34.936249 30226 raft_consensus.cc:3037] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:34.937106 30366 raft_consensus.cc:2454] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:34.937106 30226 raft_consensus.cc:2454] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:34.937278 30439 leader_election.cc:302] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a3345eeef4431f9b6229b1d75f26c2, c5180e4badca40c69f2fbafd4d75828c; no voters:
I20260925 19:09:34.937382 30583 raft_consensus.cc:2781] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:34.937554 30583 raft_consensus.cc:686] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 LEADER]: Becoming Leader. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Running, Role: LEADER
I20260925 19:09:34.937667 30583 consensus_queue.cc:237] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } }
I20260925 19:09:34.938506 30068 catalog_manager.cc:5495] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } health_report { overall_health: UNKNOWN } } }
Sep 25 19:09:34 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:34.963065 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.956254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.27.174.126:40645)
0925 19:09:34.956334 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:34.956336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:34.956439 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:09:34.956515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:34.956516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:34.956526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:34.956593 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:34.957059 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.957531 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.958226 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.958413 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.958493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.960696 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:34.960698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:34.960698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:34.960704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:34.962142 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.962476 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.962476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.962477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.962524 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:34.962684 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:34.962685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:34.962685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:34.962781 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:34.962829 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:34.962942 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:09:34.963004 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:09:34.963143 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:34.956189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40645 (local address 127.0.0.1:56642)
0925 19:09:34.956361 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:34.956371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:34.956457 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:34.956597 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:34.956597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:34.956931 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:34.956973 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.956976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.957621 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.957622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.958142 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:34.958144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:34.958493 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:34.958494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:34.958544 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:34.959059 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:34.959064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:34.960602 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:34.962214 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.962220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.962222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.962410 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.962579 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:34.962580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:34.962580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:34.962632 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:34.962833 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:34.962838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:34.962879 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 19:09:34.962929 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:09:34.963067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":54,"thread_start_us":101,"threads_started":1}
I20260925 19:09:34.964926 30068 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56642:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:09:34.965138 30068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:34.966564 30206 tablet_service.cc:1467] Processing CreateTablet for tablet 69186b39a54549f39819a376018fb4b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=958c0a417d1545b0826ba882eb7760b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:34.966601 30346 tablet_service.cc:1467] Processing CreateTablet for tablet 69186b39a54549f39819a376018fb4b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=958c0a417d1545b0826ba882eb7760b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:34.966754 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69186b39a54549f39819a376018fb4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.966759 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69186b39a54549f39819a376018fb4b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 19:09:34.967171 30273 tablet.cc:2369] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:34.968139 30575 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: Bootstrap starting.
I20260925 19:09:34.968137 30578 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: Bootstrap starting.
I20260925 19:09:34.968248 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 69186b39a54549f39819a376018fb4b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=958c0a417d1545b0826ba882eb7760b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:34.968382 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69186b39a54549f39819a376018fb4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:34.968721 30578 tablet_bootstrap.cc:654] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.968823 30575 tablet_bootstrap.cc:654] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.969347 30578 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: No bootstrap required, opened a new log
I20260925 19:09:34.969408 30578 ts_tablet_manager.cc:1397] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:34.969596 30578 raft_consensus.cc:348] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.969688 30578 raft_consensus.cc:374] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.969723 30578 raft_consensus.cc:729] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5180e4badca40c69f2fbafd4d75828c, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.969782 30578 consensus_queue.cc:260] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.969884 30578 ts_tablet_manager.cc:1428] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.969986 30577 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: Bootstrap starting.
I20260925 19:09:34.970065 30582 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:34.970165 30582 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:34.970469 30577 tablet_bootstrap.cc:654] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:34.970650 30575 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: No bootstrap required, opened a new log
I20260925 19:09:34.970695 30575 ts_tablet_manager.cc:1397] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:09:34.970824 30575 raft_consensus.cc:348] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.970870 30575 raft_consensus.cc:374] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.970882 30575 raft_consensus.cc:729] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31204205fc034c039266928d1216782e, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.970963 30575 consensus_queue.cc:260] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.971050 30575 ts_tablet_manager.cc:1428] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:34.971104 30581 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:34.971187 30581 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:34.971276 30577 tablet_bootstrap.cc:492] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: No bootstrap required, opened a new log
I20260925 19:09:34.971321 30577 ts_tablet_manager.cc:1397] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:34.971460 30577 raft_consensus.cc:348] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.971547 30577 raft_consensus.cc:374] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:34.971568 30577 raft_consensus.cc:729] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Initialized, Role: FOLLOWER
I20260925 19:09:34.971635 30577 consensus_queue.cc:260] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:34.971725 30577 ts_tablet_manager.cc:1428] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:34.971838 30583 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:34.971921 30583 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 19:09:35.034199 30553 tablet.cc:2369] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:35.123831 30583 raft_consensus.cc:482] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:35.123944 30583 raft_consensus.cc:504] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.124126 30583 leader_election.cc:288] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31204205fc034c039266928d1216782e (127.27.174.65:44983), c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823)
I20260925 19:09:35.124360 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69186b39a54549f39819a376018fb4b3" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c" is_pre_election: true
I20260925 19:09:35.124401 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69186b39a54549f39819a376018fb4b3" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e" is_pre_election: true
I20260925 19:09:35.124459 30366 raft_consensus.cc:2454] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:35.124505 30226 raft_consensus.cc:2454] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:35.124647 30437 leader_election.cc:302] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, 43a3345eeef4431f9b6229b1d75f26c2; no voters:
I20260925 19:09:35.124778 30583 raft_consensus.cc:2781] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:35.124820 30583 raft_consensus.cc:482] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:35.124852 30583 raft_consensus.cc:3037] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.125504 30583 raft_consensus.cc:504] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.125643 30583 leader_election.cc:288] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Requested vote from peers 31204205fc034c039266928d1216782e (127.27.174.65:44983), c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823)
I20260925 19:09:35.125792 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69186b39a54549f39819a376018fb4b3" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e"
I20260925 19:09:35.125864 30226 raft_consensus.cc:3037] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.125825 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69186b39a54549f39819a376018fb4b3" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c"
I20260925 19:09:35.125912 30366 raft_consensus.cc:3037] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.126458 30226 raft_consensus.cc:2454] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:35.126593 30366 raft_consensus.cc:2454] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:35.126645 30437 leader_election.cc:302] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, 43a3345eeef4431f9b6229b1d75f26c2; no voters:
I20260925 19:09:35.126760 30583 raft_consensus.cc:2781] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:35.126823 30583 raft_consensus.cc:686] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 LEADER]: Becoming Leader. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Running, Role: LEADER
I20260925 19:09:35.126868 30583 consensus_queue.cc:237] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.127193 30585 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: TxnStatusTablet state changed. Reason: New leader 43a3345eeef4431f9b6229b1d75f26c2. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.127271 30585 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:35.127467 30597 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:09:35.127584 30597 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:09:35.127794 30069 catalog_manager.cc:5495] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } health_report { overall_health: HEALTHY } } }
I20260925 19:09:35.153770 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.150888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.27.174.67:46825)
0925 19:09:35.150930 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.150932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.150939 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:35.150969 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.150970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.150986 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.151046 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.151211 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.151756 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.152881 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.153113 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.153202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.153402 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.153501 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.153568 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.153647 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.153671 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:35.153786 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.150636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46825 (local address 127.0.0.1:54772)
0925 19:09:35.150722 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.150733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.150785 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.151077 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.151078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.151084 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:35.151139 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.151142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.151839 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.151840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.152770 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.152773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.153215 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.153216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.153283 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.153303 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.153577 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.153586 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.153643 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.153693 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 19:09:35.155678 30366 raft_consensus.cc:1261] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.155731 30226 raft_consensus.cc:1261] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.156005 30583 consensus_queue.cc:1035] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.156112 30585 consensus_queue.cc:1035] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.157339 30581 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: TxnStatusTablet state changed. Reason: New leader 43a3345eeef4431f9b6229b1d75f26c2. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.157348 30582 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: TxnStatusTablet state changed. Reason: New leader 43a3345eeef4431f9b6229b1d75f26c2. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.157433 30582 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:35.157433 30581 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:35.158784 30583 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.158878 30583 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:35.158856 30582 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.158943 30582 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P c5180e4badca40c69f2fbafd4d75828c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:35.158984 30602 mvcc.cc:205] Tried to move back new op lower bound from 7333328384634695680 to 7333328384519938048. Current Snapshot: MvccSnapshot[applied={T|T < 7333328384634695680}]
I20260925 19:09:35.159287 30585 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.159387 30585 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 43a3345eeef4431f9b6229b1d75f26c2: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:35.159775 30581 tablet_replica.cc:439] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } } }
I20260925 19:09:35.159878 30581 tablet_replica.cc:442] T 69186b39a54549f39819a376018fb4b3 P 31204205fc034c039266928d1216782e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:35.159298 30600 mvcc.cc:205] Tried to move back new op lower bound from 7333328384634695680 to 7333328384519938048. Current Snapshot: MvccSnapshot[applied={T|T < 7333328384634695680}]
I20260925 19:09:35.160157 30601 mvcc.cc:205] Tried to move back new op lower bound from 7333328384634695680 to 7333328384519938048. Current Snapshot: MvccSnapshot[applied={T|T < 7333328384634695680}]
I20260925 19:09:35.164992 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.162374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.27.174.67:46825)
0925 19:09:35.162417 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.162419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.162477 (+ 58us) server_negotiation.cc:407] Connection header received
0925 19:09:35.162561 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.162562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.162573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.162634 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.162769 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.163277 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.164207 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.164443 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.164535 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.164715 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.164752 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.164824 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.164889 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.164911 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:35.170552 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.167755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46825 (local address 127.27.174.67:43391)
0925 19:09:35.167832 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.167844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.167886 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.168029 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.168029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.168031 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:35.168088 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.168089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.168944 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.168944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.169659 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.169662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.170141 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.170147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.170198 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.170210 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.170420 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.170422 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.170459 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.170492 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:09:35.170560 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.167838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:43391 (local address 127.27.174.67:46825)
0925 19:09:35.167877 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.167879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.167887 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:35.167953 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.167954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.167962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.168026 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.168146 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.168859 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.169732 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.170069 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.170143 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.170331 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.170366 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.170421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.170462 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.170481 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:35.173195 30226 raft_consensus.cc:1261] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.173252 30366 raft_consensus.cc:1261] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.173375 30583 consensus_queue.cc:1035] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.173463 30585 consensus_queue.cc:1035] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.185197 30068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:35.185320 30068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:35.186084 30068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Sending DeleteTablet for 3 replicas of tablet 08da33698184473782504656af440a9a
I20260925 19:09:35.186420 30206 tablet_service.cc:1514] Processing DeleteTablet for tablet 08da33698184473782504656af440a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52944
I20260925 19:09:35.186491 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet 08da33698184473782504656af440a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52612
I20260925 19:09:35.186513 30485 tablet_service.cc:1514] Processing DeleteTablet for tablet 08da33698184473782504656af440a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54760
I20260925 19:09:35.186714 30616 tablet_replica.cc:330] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: stopping tablet replica
I20260925 19:09:35.186821 30616 raft_consensus.cc:2229] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:35.186802 30068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:09:35.186908 30616 raft_consensus.cc:2258] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:09:35.186939 30068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:35.187135 30617 tablet_replica.cc:330] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: stopping tablet replica
I20260925 19:09:35.187234 30617 raft_consensus.cc:2229] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:35.187341 30617 raft_consensus.cc:2258] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:35.187482 30615 tablet_replica.cc:330] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: stopping tablet replica
I20260925 19:09:35.187580 30615 raft_consensus.cc:2229] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:35.187696 30615 raft_consensus.cc:2258] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:35.187938 30617 ts_tablet_manager.cc:1905] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:35.188336 30206 tablet_service.cc:1467] Processing CreateTablet for tablet b6bdb684b1864ef5b234b3940eced02d (DEFAULT_TABLE table=test-table [id=a0a49d0f67a24d878e2481705cece447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:35.188340 30346 tablet_service.cc:1467] Processing CreateTablet for tablet b6bdb684b1864ef5b234b3940eced02d (DEFAULT_TABLE table=test-table [id=a0a49d0f67a24d878e2481705cece447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:35.188454 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bdb684b1864ef5b234b3940eced02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:35.188455 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bdb684b1864ef5b234b3940eced02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:35.189757 30578 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Bootstrap starting.
I20260925 19:09:35.189765 30575 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Bootstrap starting.
I20260925 19:09:35.190153 30578 tablet_bootstrap.cc:654] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:35.190204 30575 tablet_bootstrap.cc:654] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:35.190339 30617 ts_tablet_manager.cc:1918] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:35.190454 30617 log.cc:1196] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/08da33698184473782504656af440a9a
I20260925 19:09:35.190141 30616 ts_tablet_manager.cc:1905] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:35.190697 30485 tablet_service.cc:1467] Processing CreateTablet for tablet b6bdb684b1864ef5b234b3940eced02d (DEFAULT_TABLE table=test-table [id=a0a49d0f67a24d878e2481705cece447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:35.190812 30485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bdb684b1864ef5b234b3940eced02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:35.190892 30617 ts_tablet_manager.cc:1939] T 08da33698184473782504656af440a9a P 43a3345eeef4431f9b6229b1d75f26c2: Deleting consensus metadata
I20260925 19:09:35.191977 30616 ts_tablet_manager.cc:1918] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:35.192054 30616 log.cc:1196] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/08da33698184473782504656af440a9a
I20260925 19:09:35.192091 30577 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Bootstrap starting.
I20260925 19:09:35.192337 30616 ts_tablet_manager.cc:1939] T 08da33698184473782504656af440a9a P c5180e4badca40c69f2fbafd4d75828c: Deleting consensus metadata
I20260925 19:09:35.192474 30577 tablet_bootstrap.cc:654] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:35.192761 30055 catalog_manager.cc:4841] TS c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823): tablet 08da33698184473782504656af440a9a (table test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]) successfully deleted
I20260925 19:09:35.192816 30053 catalog_manager.cc:4841] TS 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825): tablet 08da33698184473782504656af440a9a (table test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]) successfully deleted
I20260925 19:09:35.192947 30578 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: No bootstrap required, opened a new log
I20260925 19:09:35.192992 30578 ts_tablet_manager.cc:1397] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:09:35.193130 30578 raft_consensus.cc:348] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.193215 30578 raft_consensus.cc:374] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:35.193241 30578 raft_consensus.cc:729] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5180e4badca40c69f2fbafd4d75828c, State: Initialized, Role: FOLLOWER
I20260925 19:09:35.193277 30615 ts_tablet_manager.cc:1905] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:35.193281 30578 consensus_queue.cc:260] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.193372 30578 ts_tablet_manager.cc:1428] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:35.194748 30615 ts_tablet_manager.cc:1918] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:35.194819 30615 log.cc:1196] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/08da33698184473782504656af440a9a
I20260925 19:09:35.194896 30575 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: No bootstrap required, opened a new log
I20260925 19:09:35.194936 30575 ts_tablet_manager.cc:1397] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 19:09:35.195083 30575 raft_consensus.cc:348] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.195169 30575 raft_consensus.cc:374] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:35.195232 30575 raft_consensus.cc:729] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31204205fc034c039266928d1216782e, State: Initialized, Role: FOLLOWER
I20260925 19:09:35.195282 30575 consensus_queue.cc:260] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.195387 30575 ts_tablet_manager.cc:1428] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:35.195078 30615 ts_tablet_manager.cc:1939] T 08da33698184473782504656af440a9a P 31204205fc034c039266928d1216782e: Deleting consensus metadata
I20260925 19:09:35.195828 30053 catalog_manager.cc:4841] TS 31204205fc034c039266928d1216782e (127.27.174.65:44983): tablet 08da33698184473782504656af440a9a (table test-table [id=ae7e1b7b1287448a8d04bf2f4dd98595]) successfully deleted
I20260925 19:09:35.195966 30577 tablet_bootstrap.cc:492] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: No bootstrap required, opened a new log
I20260925 19:09:35.196002 30577 ts_tablet_manager.cc:1397] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 19:09:35.196099 30577 raft_consensus.cc:348] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.196141 30577 raft_consensus.cc:374] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:35.196154 30577 raft_consensus.cc:729] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Initialized, Role: FOLLOWER
I20260925 19:09:35.196214 30577 consensus_queue.cc:260] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.196298 30577 ts_tablet_manager.cc:1428] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:35.284977 30581 raft_consensus.cc:482] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:35.285099 30581 raft_consensus.cc:504] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.285496 30581 leader_election.cc:288] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825)
I20260925 19:09:35.288967 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.285828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:37413 (local address 127.27.174.67:46825)
0925 19:09:35.285900 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.285901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.285914 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:09:35.286022 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.286023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.286033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.286085 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:35.286315 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.287043 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.288144 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.288598 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.288667 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.288677 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.288859 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.288897 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 19:09:35.288970 30621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.285690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46825 (local address 127.27.174.65:37413)
0925 19:09:35.285868 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.285879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.285929 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.286108 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.286109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.286111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:35.286215 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.286217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.287132 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.287133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.288051 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.288054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.288700 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.288701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.288775 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.288780 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.288837 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.288875 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":41,"thread_start_us":87,"threads_started":1}
I20260925 19:09:35.289292 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bdb684b1864ef5b234b3940eced02d" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a3345eeef4431f9b6229b1d75f26c2" is_pre_election: true
I20260925 19:09:35.289371 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.285731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42155 (local address 127.27.174.66:44823)
0925 19:09:35.285787 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.285789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.285891 (+ 102us) server_negotiation.cc:407] Connection header received
0925 19:09:35.285984 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.285985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.285995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.286046 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:35.286287 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.287026 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.288450 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.288989 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.289058 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.289069 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.289271 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.289311 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:35.289438 30506 raft_consensus.cc:2454] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 0.
I20260925 19:09:35.289602 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.285634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44823 (local address 127.27.174.65:42155)
0925 19:09:35.285826 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.285838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.285922 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.286095 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.286096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.286098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:35.286178 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.286180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.287133 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.287134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.288355 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.288358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.289106 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.289107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.289186 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.289191 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.289474 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.289518 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260925 19:09:35.289614 30157 leader_election.cc:302] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, 43a3345eeef4431f9b6229b1d75f26c2; no voters:
I20260925 19:09:35.289814 30581 raft_consensus.cc:2781] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:35.289863 30581 raft_consensus.cc:482] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:35.289887 30581 raft_consensus.cc:3037] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.289909 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bdb684b1864ef5b234b3940eced02d" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c" is_pre_election: true
I20260925 19:09:35.289997 30366 raft_consensus.cc:2454] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 0.
I20260925 19:09:35.290655 30581 raft_consensus.cc:504] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.290829 30581 leader_election.cc:288] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 election: Requested vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825)
I20260925 19:09:35.290977 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bdb684b1864ef5b234b3940eced02d" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c"
I20260925 19:09:35.291049 30366 raft_consensus.cc:3037] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.291067 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bdb684b1864ef5b234b3940eced02d" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a3345eeef4431f9b6229b1d75f26c2"
I20260925 19:09:35.291153 30506 raft_consensus.cc:3037] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:35.291683 30366 raft_consensus.cc:2454] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 1.
I20260925 19:09:35.291867 30159 leader_election.cc:302] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, c5180e4badca40c69f2fbafd4d75828c; no voters:
I20260925 19:09:35.291939 30581 raft_consensus.cc:2781] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:35.291952 30506 raft_consensus.cc:2454] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 1.
I20260925 19:09:35.292119 30581 raft_consensus.cc:686] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 1 LEADER]: Becoming Leader. State: Replica: 31204205fc034c039266928d1216782e, State: Running, Role: LEADER
I20260925 19:09:35.292197 30581 consensus_queue.cc:237] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:35.292882 30068 catalog_manager.cc:5495] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31204205fc034c039266928d1216782e (127.27.174.65). New cstate: current_term: 1 leader_uuid: "31204205fc034c039266928d1216782e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:35.313386 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.310883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.27.174.65:44983)
0925 19:09:35.310922 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.310924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.310933 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:35.311016 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.311016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.311037 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.311134 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.311264 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.311852 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.312582 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.312806 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.312883 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.313025 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.313152 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.313239 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.313279 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.313299 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:35.318543 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.315964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:50705 (local address 127.27.174.67:46825)
0925 19:09:35.316007 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.316009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.316170 (+ 161us) server_negotiation.cc:407] Connection header received
0925 19:09:35.316217 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.316218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.316226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.316279 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.316426 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.316892 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.317928 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.318104 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.318158 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.318353 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.318377 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.318413 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.318480 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.318493 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:35.318611 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.315912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46825 (local address 127.27.174.65:50705)
0925 19:09:35.316077 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.316089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.316162 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.316297 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.316298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.316300 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:35.316364 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.316366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.316966 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.316967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.317838 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.317840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.318201 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.318202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.318262 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.318277 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.318437 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.318438 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.318480 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.318519 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260925 19:09:35.321241 30366 raft_consensus.cc:1261] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Refusing update from remote peer 31204205fc034c039266928d1216782e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.321317 30506 raft_consensus.cc:1261] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Refusing update from remote peer 31204205fc034c039266928d1216782e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:35.321513 30623 consensus_queue.cc:1035] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.321622 30581 consensus_queue.cc:1035] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:35.330686 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.327463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44983 (local address 127.27.174.67:33721)
0925 19:09:35.327548 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:35.327561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:35.327621 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:35.327947 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:35.327948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:35.327949 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:35.327985 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.327986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.328887 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.328888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.329864 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:35.329867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.330327 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.330328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:35.330373 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.330377 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.330554 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.330555 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:35.330593 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 19:09:35.330629 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 19:09:35.330685 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:35.327701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33721 (local address 127.27.174.65:44983)
0925 19:09:35.327741 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:35.327743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:35.327750 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:35.327772 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:35.327773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:35.327784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:35.327843 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:35.328066 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.328780 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.329949 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:35.330245 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:35.330324 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:35.330457 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:35.330490 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:35.330546 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:35.330597 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:09:35.330617 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:09:35.331004 30601 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:09:36.855814 30069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:36.855911 30069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:36.856743 30069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Sending DeleteTablet for 3 replicas of tablet b6bdb684b1864ef5b234b3940eced02d
I20260925 19:09:36.856994 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bdb684b1864ef5b234b3940eced02d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52612
I20260925 19:09:36.857026 30206 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bdb684b1864ef5b234b3940eced02d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52944
I20260925 19:09:36.857023 30485 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bdb684b1864ef5b234b3940eced02d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54760
I20260925 19:09:36.859182 30656 tablet_replica.cc:330] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: stopping tablet replica
I20260925 19:09:36.859279 30656 raft_consensus.cc:2229] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:36.859319 30656 raft_consensus.cc:2258] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.859485 30658 tablet_replica.cc:330] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: stopping tablet replica
I20260925 19:09:36.859560 30656 ts_tablet_manager.cc:1905] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:36.859613 30657 tablet_replica.cc:330] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: stopping tablet replica
I20260925 19:09:36.859674 30657 raft_consensus.cc:2229] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:36.859730 30657 raft_consensus.cc:2258] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.859817 30658 raft_consensus.cc:2229] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:36.859918 30658 raft_consensus.cc:2258] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.860152 30658 ts_tablet_manager.cc:1905] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:36.861006 30657 ts_tablet_manager.cc:1905] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:36.861289 30656 ts_tablet_manager.cc:1918] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:09:36.861359 30656 log.cc:1196] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/b6bdb684b1864ef5b234b3940eced02d
I20260925 19:09:36.861567 30656 ts_tablet_manager.cc:1939] T b6bdb684b1864ef5b234b3940eced02d P c5180e4badca40c69f2fbafd4d75828c: Deleting consensus metadata
I20260925 19:09:36.861893 30055 catalog_manager.cc:4841] TS c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823): tablet b6bdb684b1864ef5b234b3940eced02d (table test-table [id=a0a49d0f67a24d878e2481705cece447]) successfully deleted
I20260925 19:09:36.861990 30658 ts_tablet_manager.cc:1918] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:09:36.862051 30658 log.cc:1196] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/b6bdb684b1864ef5b234b3940eced02d
I20260925 19:09:36.862295 30658 ts_tablet_manager.cc:1939] T b6bdb684b1864ef5b234b3940eced02d P 31204205fc034c039266928d1216782e: Deleting consensus metadata
I20260925 19:09:36.862695 30657 ts_tablet_manager.cc:1918] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:09:36.862757 30657 log.cc:1196] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/b6bdb684b1864ef5b234b3940eced02d
I20260925 19:09:36.862875 30053 catalog_manager.cc:4841] TS 31204205fc034c039266928d1216782e (127.27.174.65:44983): tablet b6bdb684b1864ef5b234b3940eced02d (table test-table [id=a0a49d0f67a24d878e2481705cece447]) successfully deleted
I20260925 19:09:36.863004 30657 ts_tablet_manager.cc:1939] T b6bdb684b1864ef5b234b3940eced02d P 43a3345eeef4431f9b6229b1d75f26c2: Deleting consensus metadata
I20260925 19:09:36.863301 30053 catalog_manager.cc:4841] TS 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825): tablet b6bdb684b1864ef5b234b3940eced02d (table test-table [id=a0a49d0f67a24d878e2481705cece447]) successfully deleted
Sep 25 19:09:36 dist-test-slave-fcdn krb5kdc[30024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:09:36 dist-test-slave-fcdn krb5kdc[30024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:36.893224 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:36.875229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.27.174.126:40645)
0925 19:09:36.875409 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:36.875411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:36.875436 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:09:36.875461 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:36.875461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:36.875470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:36.875523 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:36.876133 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:36.876653 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:36.877397 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:36.877593 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:36.877649 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:36.890436 (+ 12787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:36.890438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:36.890438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:36.890447 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:36.892104 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:36.892459 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:36.892460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:36.892460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:36.892517 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:36.892703 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:36.892703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:36.892703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:36.892832 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:36.892911 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:36.893054 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:09:36.893121 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260925 19:09:36.894780 30069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:36.894927 30069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:36.896410 30346 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 (DEFAULT_TABLE table=test-table [id=61accb4980a34f75a3fb25990988e7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.896427 30485 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 (DEFAULT_TABLE table=test-table [id=61accb4980a34f75a3fb25990988e7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.896428 30206 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 (DEFAULT_TABLE table=test-table [id=61accb4980a34f75a3fb25990988e7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.896557 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3a31010a9d41668ea20fc05681fb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.896559 30485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3a31010a9d41668ea20fc05681fb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.896566 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3a31010a9d41668ea20fc05681fb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.897576 30667 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Bootstrap starting.
I20260925 19:09:36.897954 30668 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Bootstrap starting.
I20260925 19:09:36.898064 30667 tablet_bootstrap.cc:654] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.898247 30669 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Bootstrap starting.
I20260925 19:09:36.898344 30668 tablet_bootstrap.cc:654] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.898597 30669 tablet_bootstrap.cc:654] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.898669 30667 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: No bootstrap required, opened a new log
I20260925 19:09:36.898721 30667 ts_tablet_manager.cc:1397] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.898880 30668 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: No bootstrap required, opened a new log
I20260925 19:09:36.898939 30668 ts_tablet_manager.cc:1397] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.898890 30667 raft_consensus.cc:348] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.898986 30667 raft_consensus.cc:374] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.899016 30667 raft_consensus.cc:729] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5180e4badca40c69f2fbafd4d75828c, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.899065 30667 consensus_queue.cc:260] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.899120 30668 raft_consensus.cc:348] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.899196 30668 raft_consensus.cc:374] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.899219 30668 raft_consensus.cc:729] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.899257 30667 ts_tablet_manager.cc:1428] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:36.899281 30669 tablet_bootstrap.cc:492] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: No bootstrap required, opened a new log
I20260925 19:09:36.899297 30668 consensus_queue.cc:260] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.899386 30668 ts_tablet_manager.cc:1428] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:36.899544 30669 ts_tablet_manager.cc:1397] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.899690 30669 raft_consensus.cc:348] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.899767 30669 raft_consensus.cc:374] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.899791 30669 raft_consensus.cc:729] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31204205fc034c039266928d1216782e, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.899981 30669 consensus_queue.cc:260] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.900189 30669 ts_tablet_manager.cc:1428] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:09:36.914019 30630 raft_consensus.cc:482] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:36.914105 30630 raft_consensus.cc:504] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.914266 30630 leader_election.cc:288] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825)
I20260925 19:09:36.914516 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3a31010a9d41668ea20fc05681fb85" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c" is_pre_election: true
I20260925 19:09:36.914494 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3a31010a9d41668ea20fc05681fb85" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a3345eeef4431f9b6229b1d75f26c2" is_pre_election: true
I20260925 19:09:36.914618 30366 raft_consensus.cc:2454] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 0.
I20260925 19:09:36.914618 30506 raft_consensus.cc:2454] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 0.
I20260925 19:09:36.914811 30159 leader_election.cc:302] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, c5180e4badca40c69f2fbafd4d75828c; no voters:
I20260925 19:09:36.914958 30630 raft_consensus.cc:2781] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:36.914999 30630 raft_consensus.cc:482] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:36.915024 30630 raft_consensus.cc:3037] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:36.915766 30630 raft_consensus.cc:504] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.915903 30630 leader_election.cc:288] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 election: Requested vote from peers c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823), 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825)
I20260925 19:09:36.916035 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3a31010a9d41668ea20fc05681fb85" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c"
I20260925 19:09:36.916049 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3a31010a9d41668ea20fc05681fb85" candidate_uuid: "31204205fc034c039266928d1216782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a3345eeef4431f9b6229b1d75f26c2"
I20260925 19:09:36.916111 30366 raft_consensus.cc:3037] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:36.916115 30506 raft_consensus.cc:3037] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:36.916838 30366 raft_consensus.cc:2454] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 1.
I20260925 19:09:36.916838 30506 raft_consensus.cc:2454] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31204205fc034c039266928d1216782e in term 1.
I20260925 19:09:36.917035 30157 leader_election.cc:302] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31204205fc034c039266928d1216782e, 43a3345eeef4431f9b6229b1d75f26c2; no voters:
I20260925 19:09:36.917135 30630 raft_consensus.cc:2781] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:36.917191 30630 raft_consensus.cc:686] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 1 LEADER]: Becoming Leader. State: Replica: 31204205fc034c039266928d1216782e, State: Running, Role: LEADER
I20260925 19:09:36.917264 30630 consensus_queue.cc:237] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.917872 30069 catalog_manager.cc:5495] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31204205fc034c039266928d1216782e (127.27.174.65). New cstate: current_term: 1 leader_uuid: "31204205fc034c039266928d1216782e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:36.924901 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:36.922664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.27.174.65:44983)
0925 19:09:36.922803 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:36.922805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:36.922813 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:36.922834 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:36.922835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:36.922853 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:36.922910 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:36.923046 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:36.923534 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:36.924291 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:36.924479 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:36.924535 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:36.924665 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:36.924690 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:36.924726 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:36.924826 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 19:09:36.924847 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":100,"threads_started":1}
I20260925 19:09:36.926996 30506 raft_consensus.cc:1261] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Refusing update from remote peer 31204205fc034c039266928d1216782e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:36.927018 30366 raft_consensus.cc:1261] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Refusing update from remote peer 31204205fc034c039266928d1216782e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:36.927225 30651 consensus_queue.cc:1035] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:36.927304 30630 consensus_queue.cc:1035] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:36.936468 30069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:36.936563 30069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:36.937139 30069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Sending DeleteTablet for 3 replicas of tablet 9a3a31010a9d41668ea20fc05681fb85
I20260925 19:09:36.937405 30206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52944
I20260925 19:09:36.937405 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52612
I20260925 19:09:36.937423 30485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3a31010a9d41668ea20fc05681fb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:36 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54760
I20260925 19:09:36.937495 30658 tablet_replica.cc:330] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: stopping tablet replica
I20260925 19:09:36.937505 30656 tablet_replica.cc:330] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: stopping tablet replica
I20260925 19:09:36.937517 30657 tablet_replica.cc:330] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: stopping tablet replica
I20260925 19:09:36.937541 30658 raft_consensus.cc:2229] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:36.937564 30656 raft_consensus.cc:2229] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:36.937570 30657 raft_consensus.cc:2229] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:36.937588 30658 raft_consensus.cc:2258] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.937606 30656 raft_consensus.cc:2258] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.937611 30657 raft_consensus.cc:2258] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:36.937969 30069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:09:36.938035 30658 ts_tablet_manager.cc:1905] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:36.938066 30657 ts_tablet_manager.cc:1905] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:36.938100 30656 ts_tablet_manager.cc:1905] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 19:09:36.938113 30069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:36.939165 30346 tablet_service.cc:1467] Processing CreateTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 (DEFAULT_TABLE table=test-table [id=94fc145210c7457d9ab4fd91fa8aa5fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.939261 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c654c704e6dd428abd5ff28e2ff8b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.939646 30658 ts_tablet_manager.cc:1918] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:36.939702 30658 log.cc:1196] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/9a3a31010a9d41668ea20fc05681fb85
I20260925 19:09:36.939711 30657 ts_tablet_manager.cc:1918] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:36.939772 30657 log.cc:1196] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/9a3a31010a9d41668ea20fc05681fb85
I20260925 19:09:36.939927 30658 ts_tablet_manager.cc:1939] T 9a3a31010a9d41668ea20fc05681fb85 P 31204205fc034c039266928d1216782e: Deleting consensus metadata
I20260925 19:09:36.940004 30657 ts_tablet_manager.cc:1939] T 9a3a31010a9d41668ea20fc05681fb85 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting consensus metadata
I20260925 19:09:36.940135 30206 tablet_service.cc:1467] Processing CreateTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 (DEFAULT_TABLE table=test-table [id=94fc145210c7457d9ab4fd91fa8aa5fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.940167 30485 tablet_service.cc:1467] Processing CreateTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 (DEFAULT_TABLE table=test-table [id=94fc145210c7457d9ab4fd91fa8aa5fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:36.940209 30656 ts_tablet_manager.cc:1918] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:36.940237 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c654c704e6dd428abd5ff28e2ff8b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.940251 30656 log.cc:1196] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/9a3a31010a9d41668ea20fc05681fb85
I20260925 19:09:36.940254 30485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c654c704e6dd428abd5ff28e2ff8b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:36.940289 30053 catalog_manager.cc:4841] TS 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825): tablet 9a3a31010a9d41668ea20fc05681fb85 (table test-table [id=61accb4980a34f75a3fb25990988e7f5]) successfully deleted
I20260925 19:09:36.940455 30656 ts_tablet_manager.cc:1939] T 9a3a31010a9d41668ea20fc05681fb85 P c5180e4badca40c69f2fbafd4d75828c: Deleting consensus metadata
I20260925 19:09:36.940753 30055 catalog_manager.cc:4841] TS c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823): tablet 9a3a31010a9d41668ea20fc05681fb85 (table test-table [id=61accb4980a34f75a3fb25990988e7f5]) successfully deleted
I20260925 19:09:36.941303 30667 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Bootstrap starting.
I20260925 19:09:36.941480 30668 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Bootstrap starting.
I20260925 19:09:36.941505 30669 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e: Bootstrap starting.
I20260925 19:09:36.941767 30669 tablet_bootstrap.cc:654] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.941793 30668 tablet_bootstrap.cc:654] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.941879 30667 tablet_bootstrap.cc:654] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:36.942062 30053 catalog_manager.cc:4841] TS 31204205fc034c039266928d1216782e (127.27.174.65:44983): tablet 9a3a31010a9d41668ea20fc05681fb85 (table test-table [id=61accb4980a34f75a3fb25990988e7f5]) successfully deleted
I20260925 19:09:36.942342 30668 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: No bootstrap required, opened a new log
I20260925 19:09:36.942400 30668 ts_tablet_manager.cc:1397] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.942544 30667 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: No bootstrap required, opened a new log
I20260925 19:09:36.942509 30668 raft_consensus.cc:348] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942581 30668 raft_consensus.cc:374] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.942595 30667 ts_tablet_manager.cc:1397] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.942605 30668 raft_consensus.cc:729] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.942600 30669 tablet_bootstrap.cc:492] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e: No bootstrap required, opened a new log
I20260925 19:09:36.942663 30669 ts_tablet_manager.cc:1397] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:36.942653 30668 consensus_queue.cc:260] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942757 30668 ts_tablet_manager.cc:1428] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:36.942729 30667 raft_consensus.cc:348] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942793 30667 raft_consensus.cc:374] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.942771 30669 raft_consensus.cc:348] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942824 30667 raft_consensus.cc:729] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5180e4badca40c69f2fbafd4d75828c, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.942832 30669 raft_consensus.cc:374] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:36.942860 30669 raft_consensus.cc:729] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31204205fc034c039266928d1216782e, State: Initialized, Role: FOLLOWER
I20260925 19:09:36.942859 30667 consensus_queue.cc:260] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942905 30669 consensus_queue.cc:260] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:36.942967 30667 ts_tablet_manager.cc:1428] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:36.943001 30669 ts_tablet_manager.cc:1428] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:37.016271 30637 raft_consensus.cc:482] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:37.016360 30637 raft_consensus.cc:504] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:37.016605 30637 leader_election.cc:288] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31204205fc034c039266928d1216782e (127.27.174.65:44983), c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823)
I20260925 19:09:37.016805 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c654c704e6dd428abd5ff28e2ff8b930" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c" is_pre_election: true
I20260925 19:09:37.016826 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c654c704e6dd428abd5ff28e2ff8b930" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e" is_pre_election: true
I20260925 19:09:37.016916 30366 raft_consensus.cc:2454] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:37.016924 30226 raft_consensus.cc:2454] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 0.
I20260925 19:09:37.017081 30439 leader_election.cc:302] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a3345eeef4431f9b6229b1d75f26c2, c5180e4badca40c69f2fbafd4d75828c; no voters:
I20260925 19:09:37.017205 30637 raft_consensus.cc:2781] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:37.017241 30637 raft_consensus.cc:482] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:37.017266 30637 raft_consensus.cc:3037] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:37.017951 30637 raft_consensus.cc:504] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:37.018105 30637 leader_election.cc:288] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Requested vote from peers 31204205fc034c039266928d1216782e (127.27.174.65:44983), c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823)
I20260925 19:09:37.018246 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c654c704e6dd428abd5ff28e2ff8b930" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5180e4badca40c69f2fbafd4d75828c"
I20260925 19:09:37.018268 30226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c654c704e6dd428abd5ff28e2ff8b930" candidate_uuid: "43a3345eeef4431f9b6229b1d75f26c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31204205fc034c039266928d1216782e"
I20260925 19:09:37.018316 30366 raft_consensus.cc:3037] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:37.018347 30226 raft_consensus.cc:3037] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:37.018882 30366 raft_consensus.cc:2454] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:37.019002 30226 raft_consensus.cc:2454] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a3345eeef4431f9b6229b1d75f26c2 in term 1.
I20260925 19:09:37.019134 30439 leader_election.cc:302] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a3345eeef4431f9b6229b1d75f26c2, c5180e4badca40c69f2fbafd4d75828c; no voters:
I20260925 19:09:37.019223 30637 raft_consensus.cc:2781] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:37.019279 30637 raft_consensus.cc:686] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 LEADER]: Becoming Leader. State: Replica: 43a3345eeef4431f9b6229b1d75f26c2, State: Running, Role: LEADER
I20260925 19:09:37.019343 30637 consensus_queue.cc:237] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } }
I20260925 19:09:37.019960 30068 catalog_manager.cc:5495] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "43a3345eeef4431f9b6229b1d75f26c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a3345eeef4431f9b6229b1d75f26c2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46825 } health_report { overall_health: HEALTHY } } }
I20260925 19:09:37.039422 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:37.037269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.27.174.67:46825)
0925 19:09:37.037432 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:37.037434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:37.037440 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:09:37.037458 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:37.037459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:37.037467 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:37.037522 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:37.037679 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:37.038148 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:37.038858 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:37.039034 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:37.039081 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:37.039186 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:37.039210 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:37.039243 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:37.039301 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:09:37.039315 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260925 19:09:37.041499 30226 raft_consensus.cc:1261] T c654c704e6dd428abd5ff28e2ff8b930 P 31204205fc034c039266928d1216782e [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:37.041579 30366 raft_consensus.cc:1261] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Refusing update from remote peer 43a3345eeef4431f9b6229b1d75f26c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:37.041818 30643 consensus_queue.cc:1035] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5180e4badca40c69f2fbafd4d75828c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:37.041952 30634 consensus_queue.cc:1035] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31204205fc034c039266928d1216782e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:09:37.045872 30635 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:09:38.562146 30068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:38.562247 30068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:38.562973 30068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02818e07fedc4b96ae0be37b93c6710c: Sending DeleteTablet for 3 replicas of tablet c654c704e6dd428abd5ff28e2ff8b930
I20260925 19:09:38.563244 30485 tablet_service.cc:1514] Processing DeleteTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54760
I20260925 19:09:38.563251 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52612
I20260925 19:09:38.563360 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30141
I20260925 19:09:38.563441 30206 tablet_service.cc:1514] Processing DeleteTablet for tablet c654c704e6dd428abd5ff28e2ff8b930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52944
I20260925 19:09:38.564016 30709 tablet_replica.cc:330] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: stopping tablet replica
I20260925 19:09:38.564088 30709 raft_consensus.cc:2229] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:38.564149 30709 raft_consensus.cc:2258] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:38.564407 30709 ts_tablet_manager.cc:1905] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:38.566331 30709 ts_tablet_manager.cc:1918] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:09:38.566399 30709 log.cc:1196] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/c654c704e6dd428abd5ff28e2ff8b930
I20260925 19:09:38.566617 30709 ts_tablet_manager.cc:1939] T c654c704e6dd428abd5ff28e2ff8b930 P c5180e4badca40c69f2fbafd4d75828c: Deleting consensus metadata
I20260925 19:09:38.567049 30055 catalog_manager.cc:4841] TS c5180e4badca40c69f2fbafd4d75828c (127.27.174.66:44823): tablet c654c704e6dd428abd5ff28e2ff8b930 (table test-table [id=94fc145210c7457d9ab4fd91fa8aa5fc]) successfully deleted
I20260925 19:09:38.567808 30708 tablet_replica.cc:330] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: stopping tablet replica
I20260925 19:09:38.567896 30708 raft_consensus.cc:2229] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:38.567996 30708 raft_consensus.cc:2258] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:38.568236 30708 ts_tablet_manager.cc:1905] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:38.569558 30708 ts_tablet_manager.cc:1918] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:09:38.569622 30708 log.cc:1196] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/c654c704e6dd428abd5ff28e2ff8b930
I20260925 19:09:38.569845 30708 ts_tablet_manager.cc:1939] T c654c704e6dd428abd5ff28e2ff8b930 P 43a3345eeef4431f9b6229b1d75f26c2: Deleting consensus metadata
I20260925 19:09:38.570159 30053 catalog_manager.cc:4841] TS 43a3345eeef4431f9b6229b1d75f26c2 (127.27.174.67:46825): tablet c654c704e6dd428abd5ff28e2ff8b930 (table test-table [id=94fc145210c7457d9ab4fd91fa8aa5fc]) successfully deleted
W20260925 19:09:38.571650 30053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:44983 (error 108)
I20260925 19:09:38.571906 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30281
I20260925 19:09:38.572813 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:38.572057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44983 (local address 127.0.0.1:52976)
0925 19:09:38.572707 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:38.572768 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":60,"threads_started":1}
W20260925 19:09:38.572948 30053 catalog_manager.cc:4584] TS 31204205fc034c039266928d1216782e (127.27.174.65:44983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c654c704e6dd428abd5ff28e2ff8b930: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44983: connect: Connection refused (error 111)
I20260925 19:09:38.577333 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30421
I20260925 19:09:38.583698 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30037
2026-09-25T19:09:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5615 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:38 dist-test-slave-fcdn krb5kdc[30714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:38 dist-test-slave-fcdn krb5kdc[30714](info): set up 2 sockets
Sep 25 19:09:38 dist-test-slave-fcdn krb5kdc[30714](info): commencing operation
krb5kdc: starting...
W20260925 19:09:39.615953 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:39 dist-test-slave-fcdn krb5kdc[30714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:39.734136 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38963
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:39.817998 30727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:39.818217 30727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:39.818252 30727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:39.819380 30727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:39.819445 30727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:39.819473 30727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:39.819502 30727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:39.819530 30727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:39.821017 30727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:39.821280 30727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:39.821506 30727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:39.826356 30732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:39.826328 30733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:39.826344 30735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:39.826648 30727 server_base.cc:1034] running on GCE node
I20260925 19:09:39.826879 30727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:39.827159 30727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:39.828339 30727 hybrid_clock.cc:648] HybridClock initialized: now 1790363379828319 us; error 36 us; skew 500 ppm
Sep 25 19:09:39 dist-test-slave-fcdn krb5kdc[30714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:39.830662 30727 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:39.831092 30727 webserver.cc:460] Webserver started at http://127.27.174.126:39797/ using document root <none> and password file <none>
I20260925 19:09:39.831265 30727 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:39.831303 30727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:39.831398 30727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:39.832409 30727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "395b0c8ea4e3413daf2f6115d811d1e8"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "9c9607152b607e351b16afbedf281d0b"
server_key_iv: "50a1b6eaf867f94cf2f8a5ccecad9acb"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.832693 30727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "395b0c8ea4e3413daf2f6115d811d1e8"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "9c9607152b607e351b16afbedf281d0b"
server_key_iv: "50a1b6eaf867f94cf2f8a5ccecad9acb"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.833896 30727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:39.834647 30742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:39.834863 30727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:39.834946 30727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "395b0c8ea4e3413daf2f6115d811d1e8"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "9c9607152b607e351b16afbedf281d0b"
server_key_iv: "50a1b6eaf867f94cf2f8a5ccecad9acb"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.835000 30727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:39.843405 30727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:39.845103 30727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:39.845216 30727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:39.848944 30727 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38963
I20260925 19:09:39.849015 30794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38963 every 8 connection(s)
I20260925 19:09:39.849361 30727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:39.850188 30795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:39.852427 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30727
I20260925 19:09:39.852496 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:39.852677 28345 external_mini_cluster.cc:1428] Setting key b6bc2d3f014a541f313c8594f5023721
I20260925 19:09:39.852902 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: Bootstrap starting.
I20260925 19:09:39.853554 30795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:39.853835 30795 log.cc:824] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:39.854625 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: No bootstrap required, opened a new log
I20260925 19:09:39.856645 30795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } }
I20260925 19:09:39.856802 30795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:39.856861 30795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395b0c8ea4e3413daf2f6115d811d1e8, State: Initialized, Role: FOLLOWER
I20260925 19:09:39.856979 30795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } }
I20260925 19:09:39.857048 30795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:39.857090 30795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:39.857137 30795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:39.857739 30795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } }
I20260925 19:09:39.857839 30795 leader_election.cc:302] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395b0c8ea4e3413daf2f6115d811d1e8; no voters:
I20260925 19:09:39.858004 30795 leader_election.cc:288] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:39.858059 30800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:39.858242 30795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:39.858243 30800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 395b0c8ea4e3413daf2f6115d811d1e8, State: Running, Role: LEADER
I20260925 19:09:39.858395 30800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } }
I20260925 19:09:39.858676 30800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395b0c8ea4e3413daf2f6115d811d1e8. Latest consensus state: current_term: 1 leader_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } } }
I20260925 19:09:39.858908 30800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:39.858840 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395b0c8ea4e3413daf2f6115d811d1e8" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38963 } } }
I20260925 19:09:39.859272 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8 [sys.catalog]: This master's current role is: LEADER
Sep 25 19:09:39 dist-test-slave-fcdn krb5kdc[30714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
W20260925 19:09:39.860154 30816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:39.860227 30816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:39.860284 30809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:39.860456 30809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:39.862082 30809 catalog_manager.cc:1347] Generated new cluster ID: 6c53c2a48df6488784a96309e13da7b4
I20260925 19:09:39.862130 30809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:39.864324 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:39.853130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.27.174.126:38963)
0925 19:09:39.853313 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:39.853322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:39.853352 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:09:39.853817 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:39.853824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:39.853869 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:39.854016 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:39.854791 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:39.855726 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:39.856833 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:39.857372 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:39.857447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:39.860618 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:39.860622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:39.860623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:39.860633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:39.862731 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:39.863114 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:39.863117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:39.863117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:39.863217 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:39.863440 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:39.863440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:39.863441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:39.863735 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:39.863884 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:39.864010 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 19:09:39.864130 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260925 19:09:39.867627 30809 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:39.868284 30809 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:39.876565 30809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 395b0c8ea4e3413daf2f6115d811d1e8: Generated new TSK 0
I20260925 19:09:39.876718 30809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:39.902607 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:39.982510 30821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:39.982758 30821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:39.982795 30821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:39.984016 30821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:39.984076 30821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:39.984119 30821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:39.985558 30821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:39.985833 30821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:39.986057 30821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:39.990962 30827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:39.991001 30826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:39.990957 30829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:39.991043 30821 server_base.cc:1034] running on GCE node
I20260925 19:09:39.991381 30821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:39.991683 30821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:39.992834 30821 hybrid_clock.cc:648] HybridClock initialized: now 1790363379992817 us; error 35 us; skew 500 ppm
Sep 25 19:09:39 dist-test-slave-fcdn krb5kdc[30714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:39.995289 30821 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:39.995793 30821 webserver.cc:460] Webserver started at http://127.27.174.65:44115/ using document root <none> and password file <none>
I20260925 19:09:39.996023 30821 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:39.996090 30821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:39.996241 30821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:39.997228 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "8135953c301647028932e9e45da5e8d7"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "43ccda820fa3b4d6aa016546989813ca"
server_key_iv: "5c140fa44202b706a0412f6decdccae6"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.997586 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "8135953c301647028932e9e45da5e8d7"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "43ccda820fa3b4d6aa016546989813ca"
server_key_iv: "5c140fa44202b706a0412f6decdccae6"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.998837 30821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:39.999534 30836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:39.999739 30821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:39.999836 30821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "8135953c301647028932e9e45da5e8d7"
format_stamp: "Formatted at 2026-09-25 19:09:39 on dist-test-slave-fcdn"
server_key: "43ccda820fa3b4d6aa016546989813ca"
server_key_iv: "5c140fa44202b706a0412f6decdccae6"
server_key_version: "encryptionkey@0"
I20260925 19:09:39.999922 30821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:40.008587 30821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:40.010465 30821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:40.010636 30821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:40.010847 30821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:40.011135 30821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:40.011188 30821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.011226 30821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:40.011255 30821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.016846 30821 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44481
I20260925 19:09:40.016916 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44481 every 8 connection(s)
I20260925 19:09:40.017231 30821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:40.017453 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30821
I20260925 19:09:40.017553 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:40.017758 28345 external_mini_cluster.cc:1428] Setting key 69e6f0a825899efc802b4f6cb2b239e0
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[30714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:09:40.028264 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.018038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57783 (local address 127.27.174.126:38963)
0925 19:09:40.018102 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:40.018105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:40.018489 (+ 384us) server_negotiation.cc:407] Connection header received
0925 19:09:40.018991 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:40.018992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:40.019004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:40.019055 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:40.020340 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.020820 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.022364 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.022558 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.022612 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.025190 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:40.025191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:40.025192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:40.025200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:40.026670 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.027114 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.027115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.027115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.027171 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.027417 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.027417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.027418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.027531 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:40.027671 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:40.028017 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 19:09:40.028115 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20260925 19:09:40.028400 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.018017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38963 (local address 127.27.174.65:57783)
0925 19:09:40.018368 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:40.018417 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:40.018890 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:40.019141 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:40.019144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:40.019950 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:40.020230 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.020237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.020915 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.020916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.022277 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.022279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.022815 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.022817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.022976 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.023494 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:40.023505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:40.025091 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:40.026789 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.026795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.026797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.027028 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.027238 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.027239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.027239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.027348 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.027666 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:40.027671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:40.027886 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 19:09:40.027996 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:09:40.028143 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":80,"threads_started":1}
I20260925 19:09:40.028970 30950 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38963
I20260925 19:09:40.029059 30950 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:40.029277 30950 heartbeater.cc:511] Master 127.27.174.126:38963 requested a full tablet report, sending...
I20260925 19:09:40.029848 30759 ts_manager.cc:194] Registered new tserver with Master: 8135953c301647028932e9e45da5e8d7 (127.27.174.65:44481)
I20260925 19:09:40.030458 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:57783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:40.042676 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:40.124456 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:40.124650 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:40.124672 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:40.125830 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:40.125880 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:40.125932 30955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:40.127300 30955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:40.127535 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:40.127780 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:40.132553 30961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:40.132561 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:40.132614 30960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:40.132863 30955 server_base.cc:1034] running on GCE node
I20260925 19:09:40.133080 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:40.133311 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:40.134526 30955 hybrid_clock.cc:648] HybridClock initialized: now 1790363380134474 us; error 23 us; skew 500 ppm
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[30714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:40.137097 30955 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:40.137573 30955 webserver.cc:460] Webserver started at http://127.27.174.66:43097/ using document root <none> and password file <none>
I20260925 19:09:40.137864 30955 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:40.137933 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:40.138069 30955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:40.139174 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "70628947b25a49819014ac8c18a03187"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "d14f6f7b12dde74267769e930b10947d"
server_key_iv: "df8c741c29c3511e013c32a2c96b571c"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.139534 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "70628947b25a49819014ac8c18a03187"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "d14f6f7b12dde74267769e930b10947d"
server_key_iv: "df8c741c29c3511e013c32a2c96b571c"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.140866 30955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:40.141616 30970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.141788 30955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:40.141886 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "70628947b25a49819014ac8c18a03187"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "d14f6f7b12dde74267769e930b10947d"
server_key_iv: "df8c741c29c3511e013c32a2c96b571c"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.141966 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:40.166276 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:40.168725 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:40.168900 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:40.169154 30955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:40.169484 30955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:40.169538 30955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.169579 30955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:40.169606 30955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.175422 30955 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:46621
I20260925 19:09:40.175463 31083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:46621 every 8 connection(s)
I20260925 19:09:40.175859 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:40.178005 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 30955
I20260925 19:09:40.178110 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:40.178308 28345 external_mini_cluster.cc:1428] Setting key fb65455138f7cd684d5cb4b9213abe57
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[30714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20260925 19:09:40.186421 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.176626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:36033 (local address 127.27.174.126:38963)
0925 19:09:40.176705 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:40.176707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:40.176906 (+ 199us) server_negotiation.cc:407] Connection header received
0925 19:09:40.177365 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:40.177366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:40.177376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:40.177430 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:40.178669 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.179357 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.180848 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.181176 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.181256 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.183581 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:40.183583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:40.183583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:40.183591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:40.185033 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.185463 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.185464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.185464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.185524 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.185771 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.185772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.185773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.185890 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:40.185964 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:40.186236 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 19:09:40.186332 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:09:40.186602 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.176614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38963 (local address 127.27.174.66:36033)
0925 19:09:40.176776 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:40.176810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:40.177280 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:40.177504 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:40.177507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:40.178241 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:40.178538 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.178544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.179447 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.179448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.180758 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.180761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.181262 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.181263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.181368 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.181896 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:40.181906 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:40.183482 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:40.185139 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.185145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.185147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.185376 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.185593 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.185594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.185594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.185703 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.185966 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:40.185972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:40.186110 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 19:09:40.186218 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:09:40.186375 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260925 19:09:40.187162 31084 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38963
I20260925 19:09:40.187266 31084 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:40.187457 31084 heartbeater.cc:511] Master 127.27.174.126:38963 requested a full tablet report, sending...
I20260925 19:09:40.187889 30759 ts_manager.cc:194] Registered new tserver with Master: 70628947b25a49819014ac8c18a03187 (127.27.174.66:46621)
I20260925 19:09:40.188262 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:36033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:40.202919 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:40.281430 31089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:40.281617 31089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:40.281639 31089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:40.282768 31089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:40.282811 31089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:40.282836 31089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:40.284197 31089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:40.284413 31089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:40.284660 31089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:40.289494 31097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:40.289546 31095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:40.289649 31094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:40.289677 31089 server_base.cc:1034] running on GCE node
I20260925 19:09:40.289923 31089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:40.290148 31089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:40.291302 31089 hybrid_clock.cc:648] HybridClock initialized: now 1790363380291286 us; error 48 us; skew 500 ppm
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[30714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:40.293731 31089 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:40.294132 31089 webserver.cc:460] Webserver started at http://127.27.174.67:41327/ using document root <none> and password file <none>
I20260925 19:09:40.294346 31089 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:40.294411 31089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:40.294559 31089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:40.295583 31089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "08e9a692f2a442b9bd884aa148748a30"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "10894bb5c08d53fc13d58f338cc9e3b9"
server_key_iv: "460beadcaa06bd4262d5c4eb94a513c6"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.296015 31089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "08e9a692f2a442b9bd884aa148748a30"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "10894bb5c08d53fc13d58f338cc9e3b9"
server_key_iv: "460beadcaa06bd4262d5c4eb94a513c6"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.297287 31089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:40.297950 31104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.298121 31089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:40.298214 31089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "08e9a692f2a442b9bd884aa148748a30"
format_stamp: "Formatted at 2026-09-25 19:09:40 on dist-test-slave-fcdn"
server_key: "10894bb5c08d53fc13d58f338cc9e3b9"
server_key_iv: "460beadcaa06bd4262d5c4eb94a513c6"
server_key_version: "encryptionkey@0"
I20260925 19:09:40.298295 31089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:40.334249 31089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:40.336719 31089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:40.336886 31089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:40.337134 31089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:40.337560 31089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:40.337615 31089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.337648 31089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:40.337705 31089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:40.343436 31089 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:39253
I20260925 19:09:40.343511 31217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:39253 every 8 connection(s)
I20260925 19:09:40.343848 31089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:40.347554 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31089
I20260925 19:09:40.347682 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:40.347877 28345 external_mini_cluster.cc:1428] Setting key 3aa3619feaa779d639ffa519a6e3c993
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[30714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:40.353236 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.344557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:54725 (local address 127.27.174.126:38963)
0925 19:09:40.344688 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:40.344691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:40.344952 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:09:40.345420 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:40.345421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:40.345432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:40.345484 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:40.346485 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.346977 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.348363 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.348628 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.348678 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.350902 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:40.350903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:40.350904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:40.350910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:40.352114 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.352495 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.352496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.352496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.352541 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:40.352771 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:40.352771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:40.352771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:40.352861 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:40.352908 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:40.353124 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 19:09:40.353176 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:09:40.353422 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:40.344539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38963 (local address 127.27.174.67:54725)
0925 19:09:40.344792 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:40.344856 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:40.345338 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:40.345537 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:40.345540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:40.346199 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:40.346401 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.346405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.347078 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.347079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.348225 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:40.348227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:40.348724 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:40.348725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:40.348845 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:40.349444 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:40.349459 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:40.350822 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:40.352182 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.352187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.352189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.352428 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.352599 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:40.352599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:40.352600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:40.352716 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:40.352914 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:40.352918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:40.353023 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 19:09:40.353100 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:09:40.353214 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260925 19:09:40.353892 31218 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38963
I20260925 19:09:40.353992 31218 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:40.354152 31218 heartbeater.cc:511] Master 127.27.174.126:38963 requested a full tablet report, sending...
I20260925 19:09:40.354499 30759 ts_manager.cc:194] Registered new tserver with Master: 08e9a692f2a442b9bd884aa148748a30 (127.27.174.67:39253)
I20260925 19:09:40.354868 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:54725
I20260925 19:09:40.359840 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:09:40.370492 30798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:40.369567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.27.174.126:38963)
0925 19:09:40.369640 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:40.369642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:40.369745 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:09:40.369823 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:40.369824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:40.369834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:40.369906 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:40.370423 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36734: BlockingRecv error: recv got EOF from 127.0.0.1:36734 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:09:40.371304 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30821
I20260925 19:09:40.375974 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30955
I20260925 19:09:40.380513 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31089
I20260925 19:09:40.384459 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 30727
2026-09-25T19:09:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1799 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[31230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[31230](info): set up 2 sockets
Sep 25 19:09:40 dist-test-slave-fcdn krb5kdc[31230](info): commencing operation
krb5kdc: starting...
W20260925 19:09:41.423401 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:41.537992 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35449
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:41.616714 31243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:41.616909 31243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:41.616931 31243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:41.618207 31243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:41.618251 31243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:41.618266 31243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:41.618278 31243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:41.618332 31243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:41.619813 31243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:41.620057 31243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:41.620282 31243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:41.625066 31251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:41.625036 31249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:41.625128 31248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:41.625290 31243 server_base.cc:1034] running on GCE node
I20260925 19:09:41.625502 31243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:41.625737 31243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:41.626870 31243 hybrid_clock.cc:648] HybridClock initialized: now 1790363381626847 us; error 39 us; skew 500 ppm
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363381, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:41.629303 31243 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:41.629750 31243 webserver.cc:460] Webserver started at http://127.27.174.126:40151/ using document root <none> and password file <none>
I20260925 19:09:41.629964 31243 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:41.630028 31243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:41.630182 31243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:41.631181 31243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "50a0fe0c42c746adab1a0209c01f726b"
format_stamp: "Formatted at 2026-09-25 19:09:41 on dist-test-slave-fcdn"
server_key: "67b3ec288b956784ad38d3ff21f04c71"
server_key_iv: "12ffb989d58fb2166fe4da564eb0283b"
server_key_version: "encryptionkey@0"
I20260925 19:09:41.631546 31243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "50a0fe0c42c746adab1a0209c01f726b"
format_stamp: "Formatted at 2026-09-25 19:09:41 on dist-test-slave-fcdn"
server_key: "67b3ec288b956784ad38d3ff21f04c71"
server_key_iv: "12ffb989d58fb2166fe4da564eb0283b"
server_key_version: "encryptionkey@0"
I20260925 19:09:41.632841 31243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:41.633492 31258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:41.633649 31243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:41.633744 31243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "50a0fe0c42c746adab1a0209c01f726b"
format_stamp: "Formatted at 2026-09-25 19:09:41 on dist-test-slave-fcdn"
server_key: "67b3ec288b956784ad38d3ff21f04c71"
server_key_iv: "12ffb989d58fb2166fe4da564eb0283b"
server_key_version: "encryptionkey@0"
I20260925 19:09:41.633821 31243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:41.654142 31243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:41.656443 31243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:41.656603 31243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:41.660444 31243 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35449
I20260925 19:09:41.660490 31310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35449 every 8 connection(s)
I20260925 19:09:41.660971 31243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:41.661608 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:41.661986 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31243
I20260925 19:09:41.662076 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:41.662247 28345 external_mini_cluster.cc:1428] Setting key 4d99c602a1bf4dae8712f9d50bda665b
I20260925 19:09:41.664706 31311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b: Bootstrap starting.
I20260925 19:09:41.665403 31311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:41.665693 31311 log.cc:824] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:41.666390 31311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b: No bootstrap required, opened a new log
I20260925 19:09:41.668776 31311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } }
I20260925 19:09:41.668948 31311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:41.668987 31311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a0fe0c42c746adab1a0209c01f726b, State: Initialized, Role: FOLLOWER
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:41.669112 31311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } }
I20260925 19:09:41.669224 31311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:41.669267 31311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:41.669317 31311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:41.670047 31311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } }
I20260925 19:09:41.670166 31311 leader_election.cc:302] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50a0fe0c42c746adab1a0209c01f726b; no voters:
I20260925 19:09:41.670338 31311 leader_election.cc:288] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:41.670409 31316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:41.670591 31311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:41.670576 31316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [term 1 LEADER]: Becoming Leader. State: Replica: 50a0fe0c42c746adab1a0209c01f726b, State: Running, Role: LEADER
I20260925 19:09:41.670800 31316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } }
I20260925 19:09:41.671155 31316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50a0fe0c42c746adab1a0209c01f726b. Latest consensus state: current_term: 1 leader_uuid: "50a0fe0c42c746adab1a0209c01f726b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } } }
I20260925 19:09:41.671268 31316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:41.671506 31325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:41.672317 31318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50a0fe0c42c746adab1a0209c01f726b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a0fe0c42c746adab1a0209c01f726b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35449 } } }
I20260925 19:09:41.672412 31318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:41.672595 31325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:41.673808 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:41.662723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.27.174.126:35449)
0925 19:09:41.662921 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:41.662930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:41.662964 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:09:41.663478 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:41.663486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:41.663549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:41.663736 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:41.664337 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:41.665677 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:41.666404 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:41.666978 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:41.667089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:41.669573 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:41.669577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:41.669578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:41.669589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:41.671914 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:41.672336 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:41.672338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:41.672339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:41.672438 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:41.672659 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:41.672660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:41.672661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:41.672895 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:41.673059 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:41.673458 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 19:09:41.673615 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260925 19:09:41.674140 31325 catalog_manager.cc:1347] Generated new cluster ID: e9687d797202414c85fdcd5c6b41b6bb
I20260925 19:09:41.674196 31325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:41.674611 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:41.678927 31325 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:41.679708 31325 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:41.689882 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:41.685448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.27.174.126:35449)
0925 19:09:41.685488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:41.685490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:41.685621 (+ 131us) server_negotiation.cc:407] Connection header received
0925 19:09:41.685654 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:41.685655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:41.685665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:41.685732 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:41.686243 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:41.686999 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:41.688226 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:41.688545 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:41.688616 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:41.689414 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:41.689416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:41.689416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:41.689623 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:41.689642 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:41.689649 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:41.689711 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:41.689783 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 19:09:41.689806 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:41.691264 31325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50a0fe0c42c746adab1a0209c01f726b: Generated new TSK 0
I20260925 19:09:41.691466 31325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:41.697813 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31243
2026-09-25T19:09:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1311 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31344](info): set up 2 sockets
Sep 25 19:09:41 dist-test-slave-fcdn krb5kdc[31344](info): commencing operation
krb5kdc: starting...
W20260925 19:09:42.735409 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:42 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:42.854823 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45965
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:42.933972 31357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:42.934170 31357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:42.934191 31357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:42.935261 31357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:42.935294 31357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:42.935308 31357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:42.935320 31357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:42.935333 31357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:42.936704 31357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:42.936888 31357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:42.937044 31357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:42.942005 31362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:42.942024 31363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:42.942024 31365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:42.942101 31357 server_base.cc:1034] running on GCE node
I20260925 19:09:42.942472 31357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:42.942781 31357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:42.943974 31357 hybrid_clock.cc:648] HybridClock initialized: now 1790363382943952 us; error 44 us; skew 500 ppm
Sep 25 19:09:42 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363382, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:42.956281 31357 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:42.956837 31357 webserver.cc:460] Webserver started at http://127.27.174.126:33347/ using document root <none> and password file <none>
I20260925 19:09:42.957070 31357 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:42.957137 31357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:42.957296 31357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:42.958389 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "ba5c794449e44b8e9bb90f9f1783ac3d"
format_stamp: "Formatted at 2026-09-25 19:09:42 on dist-test-slave-fcdn"
server_key: "96874c133ba288ca9828ee299027becf"
server_key_iv: "b8aa5cc032b41074b31e6bedfc1c285c"
server_key_version: "encryptionkey@0"
I20260925 19:09:42.958765 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ba5c794449e44b8e9bb90f9f1783ac3d"
format_stamp: "Formatted at 2026-09-25 19:09:42 on dist-test-slave-fcdn"
server_key: "96874c133ba288ca9828ee299027becf"
server_key_iv: "b8aa5cc032b41074b31e6bedfc1c285c"
server_key_version: "encryptionkey@0"
I20260925 19:09:42.960170 31357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:42.960913 31372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:42.961081 31357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:42.961187 31357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "ba5c794449e44b8e9bb90f9f1783ac3d"
format_stamp: "Formatted at 2026-09-25 19:09:42 on dist-test-slave-fcdn"
server_key: "96874c133ba288ca9828ee299027becf"
server_key_iv: "b8aa5cc032b41074b31e6bedfc1c285c"
server_key_version: "encryptionkey@0"
I20260925 19:09:42.961261 31357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:42.972219 31357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:42.973901 31357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:42.974056 31357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:42.978184 31357 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45965
I20260925 19:09:42.978216 31424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45965 every 8 connection(s)
I20260925 19:09:42.978554 31357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:42.979332 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31357
I20260925 19:09:42.979446 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:42.979537 31425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:42.979768 28345 external_mini_cluster.cc:1428] Setting key bcad66391188a2e0b202c403ba0d94e5
I20260925 19:09:42.982704 31425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d: Bootstrap starting.
I20260925 19:09:42.983412 31425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:42.983732 31425 log.cc:824] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:42.984453 31425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d: No bootstrap required, opened a new log
Sep 25 19:09:42 dist-test-slave-fcdn krb5kdc[31344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:42.986977 31425 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } }
I20260925 19:09:42.987130 31425 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:42.987172 31425 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5c794449e44b8e9bb90f9f1783ac3d, State: Initialized, Role: FOLLOWER
I20260925 19:09:42.987299 31425 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } }
I20260925 19:09:42.987396 31425 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:42.987439 31425 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:42.987488 31425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:42.988049 31425 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } }
I20260925 19:09:42.988165 31425 leader_election.cc:302] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5c794449e44b8e9bb90f9f1783ac3d; no voters:
I20260925 19:09:42.988363 31425 leader_election.cc:288] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:42.988395 31430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:42.988552 31430 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [term 1 LEADER]: Becoming Leader. State: Replica: ba5c794449e44b8e9bb90f9f1783ac3d, State: Running, Role: LEADER
I20260925 19:09:42.988628 31425 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:42.988742 31430 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } }
I20260925 19:09:42.989145 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } } }
I20260925 19:09:42.989285 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:42.989461 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5c794449e44b8e9bb90f9f1783ac3d. Latest consensus state: current_term: 1 leader_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5c794449e44b8e9bb90f9f1783ac3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45965 } } }
I20260925 19:09:42.989521 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:42.989660 31439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:42.990157 31439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:42.991153 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:42.980320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.27.174.126:45965)
0925 19:09:42.980493 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:42.980502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:42.980536 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:09:42.981040 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:42.981047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:42.981095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:42.981225 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:42.982017 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:42.983016 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:42.984098 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:42.984605 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:42.984686 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:42.987303 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:42.987308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:42.987308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:42.987320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:42.989372 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:42.989794 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:42.989797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:42.989797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:42.989903 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:42.990093 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:42.990093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:42.990094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:42.990405 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:42.990561 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:42.990843 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 19:09:42.990976 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260925 19:09:42.992177 31439 catalog_manager.cc:1347] Generated new cluster ID: d4a8edd5761b43e689a44eb7eaa43d7a
I20260925 19:09:42.992226 31439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:43.001536 31439 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:43.002243 31439 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:43.007282 31439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5c794449e44b8e9bb90f9f1783ac3d: Generated new TSK 0
I20260925 19:09:43.007452 31439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:43.038990 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:43.119241 31451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:43.119447 31451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:43.119470 31451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:43.120640 31451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:43.120720 31451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:43.120764 31451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:43.122227 31451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:43.122411 31451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:43.122574 31451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:43.127377 31459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.127401 31456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.127399 31457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:43.127573 31451 server_base.cc:1034] running on GCE node
I20260925 19:09:43.127830 31451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:43.128082 31451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:43.129240 31451 hybrid_clock.cc:648] HybridClock initialized: now 1790363383129217 us; error 36 us; skew 500 ppm
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:43.131752 31451 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:43.132195 31451 webserver.cc:460] Webserver started at http://127.27.174.65:45439/ using document root <none> and password file <none>
I20260925 19:09:43.132386 31451 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:43.132426 31451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:43.132514 31451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:43.133466 31451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "4f4f7c43e6fd46699077c65d5011a62d"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "4a658b9ce11db0d7970117c376893c8d"
server_key_iv: "ce32d1261fb396aeb8bb3edcfcb25210"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.133756 31451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "4f4f7c43e6fd46699077c65d5011a62d"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "4a658b9ce11db0d7970117c376893c8d"
server_key_iv: "ce32d1261fb396aeb8bb3edcfcb25210"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.134984 31451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:43.135651 31466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.135847 31451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:43.135944 31451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "4f4f7c43e6fd46699077c65d5011a62d"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "4a658b9ce11db0d7970117c376893c8d"
server_key_iv: "ce32d1261fb396aeb8bb3edcfcb25210"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.136032 31451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:43.159950 31451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:43.161850 31451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:43.162016 31451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:43.162248 31451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:43.162570 31451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:43.162624 31451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.162660 31451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:43.162689 31451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.168555 31451 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41775
I20260925 19:09:43.168649 31579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41775 every 8 connection(s)
I20260925 19:09:43.168937 31451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:43.174057 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31451
I20260925 19:09:43.174171 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:43.174379 28345 external_mini_cluster.cc:1428] Setting key 604fa1b6cb379afdbd2b3de95ca316a7
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:43.178488 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.169643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43955 (local address 127.27.174.126:45965)
0925 19:09:43.169708 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:43.169711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:43.169935 (+ 224us) server_negotiation.cc:407] Connection header received
0925 19:09:43.170391 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:43.170392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:43.170400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:43.170451 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:43.171335 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.171960 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.172892 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.173083 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.173133 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.175325 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:43.175326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:43.175327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:43.175334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:43.177127 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.177527 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.177529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.177529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.177593 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.177854 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.177855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.177856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.177968 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:43.178045 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:43.178307 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 19:09:43.178394 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:43.178668 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.169632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45965 (local address 127.27.174.65:43955)
0925 19:09:43.169779 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:43.169821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:43.170307 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:43.170504 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:43.170508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:43.171052 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:43.171237 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.171241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.172039 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.172039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.172823 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.172825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.173149 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.173150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.173251 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.173790 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:43.173798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:43.175235 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:43.177205 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.177211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.177212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.177436 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.177664 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.177665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.177666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.177778 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.178035 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:43.178042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:43.178172 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 19:09:43.178282 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:09:43.178437 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 19:09:43.179142 31580 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45965
I20260925 19:09:43.179243 31580 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:43.179447 31580 heartbeater.cc:511] Master 127.27.174.126:45965 requested a full tablet report, sending...
I20260925 19:09:43.179992 31389 ts_manager.cc:194] Registered new tserver with Master: 4f4f7c43e6fd46699077c65d5011a62d (127.27.174.65:41775)
I20260925 19:09:43.180589 31389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:43955
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:43.199854 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:43.279361 31585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:43.279547 31585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:43.279570 31585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:43.280722 31585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:43.280763 31585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:43.280789 31585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:43.282109 31585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:43.282275 31585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:43.282439 31585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:43.287237 31593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.287261 31590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.287285 31591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:43.287386 31585 server_base.cc:1034] running on GCE node
I20260925 19:09:43.287650 31585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:43.287925 31585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:43.289094 31585 hybrid_clock.cc:648] HybridClock initialized: now 1790363383289088 us; error 36 us; skew 500 ppm
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:43.291770 31585 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:43.292240 31585 webserver.cc:460] Webserver started at http://127.27.174.66:39219/ using document root <none> and password file <none>
I20260925 19:09:43.292470 31585 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:43.292536 31585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:43.292670 31585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:43.293673 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "d1bcbadac27e4386b5efad2d8f06d403"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "ab8184b98e830a615ad16a497bf9dbbd"
server_key_iv: "22df8fafa9241a7ad389dca5f23638a7"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.294034 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "d1bcbadac27e4386b5efad2d8f06d403"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "ab8184b98e830a615ad16a497bf9dbbd"
server_key_iv: "22df8fafa9241a7ad389dca5f23638a7"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.295331 31585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:09:43.296144 31600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.296315 31585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.296414 31585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "d1bcbadac27e4386b5efad2d8f06d403"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "ab8184b98e830a615ad16a497bf9dbbd"
server_key_iv: "22df8fafa9241a7ad389dca5f23638a7"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.296499 31585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:43.316571 31585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:43.318944 31585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:43.319110 31585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:43.319356 31585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:43.319675 31585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:43.319729 31585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.319766 31585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:43.319793 31585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.325598 31585 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:43317
I20260925 19:09:43.325641 31713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:43317 every 8 connection(s)
I20260925 19:09:43.325938 31585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:43.334380 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31585
I20260925 19:09:43.334506 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:43.334702 28345 external_mini_cluster.cc:1428] Setting key 81abae93a4a9204b70fb406351d3f197
I20260925 19:09:43.334901 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.326641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:54657 (local address 127.27.174.126:45965)
0925 19:09:43.326684 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:43.326686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:43.326896 (+ 210us) server_negotiation.cc:407] Connection header received
0925 19:09:43.327412 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:43.327414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:43.327425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:43.327479 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:43.328443 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.328937 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.329925 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.330110 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.330199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.332307 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:43.332309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:43.332309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:43.332320 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:43.333472 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.333877 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.333879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.333879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.333938 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.334191 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.334192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.334192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.334315 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:43.334409 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:43.334719 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 19:09:43.334805 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:43.335114 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.326617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45965 (local address 127.27.174.66:54657)
0925 19:09:43.326781 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:43.326815 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:43.327315 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:43.327547 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:43.327551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:43.328166 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:43.328347 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.328352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.329033 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.329034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.329825 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.329827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.330208 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.330209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.330335 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.330815 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:43.330824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:43.332211 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:43.333557 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.333563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.333566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.333792 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.334008 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.334009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.334009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.334122 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.334402 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:43.334408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:43.334557 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 19:09:43.334701 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 19:09:43.334868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260925 19:09:43.335556 31714 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45965
I20260925 19:09:43.335678 31714 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:43.335860 31714 heartbeater.cc:511] Master 127.27.174.126:45965 requested a full tablet report, sending...
I20260925 19:09:43.336251 31388 ts_manager.cc:194] Registered new tserver with Master: d1bcbadac27e4386b5efad2d8f06d403 (127.27.174.66:43317)
I20260925 19:09:43.336709 31388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:54657
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:43.360020 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:43.439566 31719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:43.439775 31719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:43.439798 31719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:43.440938 31719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:43.440979 31719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:43.441037 31719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:43.442453 31719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:43.442687 31719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:43.442906 31719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:43.447916 31725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.447916 31724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:43.447932 31727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:43.448205 31719 server_base.cc:1034] running on GCE node
I20260925 19:09:43.448398 31719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:43.448588 31719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:43.449795 31719 hybrid_clock.cc:648] HybridClock initialized: now 1790363383449780 us; error 23 us; skew 500 ppm
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:43.452221 31719 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:43.452584 31719 webserver.cc:460] Webserver started at http://127.27.174.67:32835/ using document root <none> and password file <none>
I20260925 19:09:43.452759 31719 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:43.452797 31719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:43.452891 31719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:43.453891 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "a9924efec36447908353188573970adb"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "aac7bcbc110acbc421304c41cc24c4bb"
server_key_iv: "8813412a0fd935e6d05a9c7dfefaeb3c"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.454183 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "a9924efec36447908353188573970adb"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "aac7bcbc110acbc421304c41cc24c4bb"
server_key_iv: "8813412a0fd935e6d05a9c7dfefaeb3c"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.455400 31719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:09:43.456126 31734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.456272 31719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.456357 31719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "a9924efec36447908353188573970adb"
format_stamp: "Formatted at 2026-09-25 19:09:43 on dist-test-slave-fcdn"
server_key: "aac7bcbc110acbc421304c41cc24c4bb"
server_key_iv: "8813412a0fd935e6d05a9c7dfefaeb3c"
server_key_version: "encryptionkey@0"
I20260925 19:09:43.456409 31719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:43.471448 31719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:43.473443 31719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:43.473552 31719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:43.473724 31719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:43.474026 31719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:43.474059 31719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.474110 31719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:43.474145 31719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:43.479743 31719 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:34237
I20260925 19:09:43.479858 31847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:34237 every 8 connection(s)
I20260925 19:09:43.480141 31719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:43.484297 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31719
I20260925 19:09:43.484419 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:43.484613 28345 external_mini_cluster.cc:1428] Setting key 80ed96963b20e1ee0b1a666be60eee91
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:43.489331 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.480882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44383 (local address 127.27.174.126:45965)
0925 19:09:43.480926 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:43.480928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:43.481125 (+ 197us) server_negotiation.cc:407] Connection header received
0925 19:09:43.481593 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:43.481594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:43.481603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:43.481660 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:43.482543 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.483032 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.484061 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.484319 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.484408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.486483 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:43.486484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:43.486485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:43.486493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:43.487868 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.488335 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.488336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.488337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.488395 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.488642 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.488643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.488643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.488792 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:43.488873 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:43.489131 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 19:09:43.489207 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:43.489490 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.480833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45965 (local address 127.27.174.67:44383)
0925 19:09:43.480996 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:43.481038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:43.481507 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:43.481713 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:43.481717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:43.482274 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:43.482464 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.482468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.483108 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.483108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.483989 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:43.483991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.484414 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.484416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:43.484578 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.485088 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:43.485096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:43.486388 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:43.487969 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.487975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.487977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.488260 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.488464 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:43.488465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:43.488466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:43.488568 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:43.488863 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:43.488869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:43.489003 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 19:09:43.489111 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:09:43.489261 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 19:09:43.490002 31848 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45965
I20260925 19:09:43.490080 31848 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:43.490208 31848 heartbeater.cc:511] Master 127.27.174.126:45965 requested a full tablet report, sending...
I20260925 19:09:43.490658 31388 ts_manager.cc:194] Registered new tserver with Master: a9924efec36447908353188573970adb (127.27.174.67:34237)
I20260925 19:09:43.491032 31388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:44383
I20260925 19:09:43.496066 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:43.517620 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:43.510958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.27.174.126:45965)
0925 19:09:43.511022 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:43.511024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:43.511145 (+ 121us) server_negotiation.cc:407] Connection header received
0925 19:09:43.511193 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:43.511194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:43.511203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:43.511252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:43.511840 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.512317 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.513007 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:43.513175 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:43.513222 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:43.515391 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:43.515393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:43.515393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:43.515400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:43.516666 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.516968 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.516968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.516969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.517016 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:43.517175 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:43.517176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:43.517176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:43.517309 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:43.517382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:43.517463 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:09:43.517520 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 19:09:43.517877 31388 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54392
I20260925 19:09:43.518312 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31451
I20260925 19:09:43.523092 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31585
I20260925 19:09:43.527159 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31719
I20260925 19:09:43.531234 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31357
2026-09-25T19:09:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1835 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31860](info): set up 2 sockets
Sep 25 19:09:43 dist-test-slave-fcdn krb5kdc[31860](info): commencing operation
krb5kdc: starting...
W20260925 19:09:44.571457 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:44 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:44.685894 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43561
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:44.764958 31873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:44.765156 31873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:44.765178 31873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:44.766325 31873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:44.766366 31873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:44.766379 31873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:44.766430 31873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:44.766458 31873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:44.768000 31873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:44.768255 31873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:44.768477 31873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:44.773190 31879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:44.773182 31881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:44.773185 31878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:44.773212 31873 server_base.cc:1034] running on GCE node
I20260925 19:09:44.773561 31873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:44.773793 31873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:44.774928 31873 hybrid_clock.cc:648] HybridClock initialized: now 1790363384774884 us; error 62 us; skew 500 ppm
Sep 25 19:09:44 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363384, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:44.777459 31873 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:44.777835 31873 webserver.cc:460] Webserver started at http://127.27.174.126:39013/ using document root <none> and password file <none>
I20260925 19:09:44.778013 31873 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:44.778053 31873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:44.778139 31873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:44.779104 31873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "1bc5c21658ad44a1b3a33d60b6c6e022"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "2c1d27afae4fd6f7dd2071493b4cc50c"
server_key_iv: "363bede1923a5ea8782e6f2796e15a4b"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.779408 31873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "1bc5c21658ad44a1b3a33d60b6c6e022"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "2c1d27afae4fd6f7dd2071493b4cc50c"
server_key_iv: "363bede1923a5ea8782e6f2796e15a4b"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.780715 31873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:44.781333 31888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:44.781491 31873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:44.781586 31873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "1bc5c21658ad44a1b3a33d60b6c6e022"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "2c1d27afae4fd6f7dd2071493b4cc50c"
server_key_iv: "363bede1923a5ea8782e6f2796e15a4b"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.781664 31873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:44.796116 31873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:44.797923 31873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:44.798089 31873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:44.801982 31873 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43561
I20260925 19:09:44.802017 31940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43561 every 8 connection(s)
I20260925 19:09:44.802534 31873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:44.803160 31941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:44.805356 31941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022: Bootstrap starting.
I20260925 19:09:44.805926 31941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:44.806213 31941 log.cc:824] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:44.806865 31941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022: No bootstrap required, opened a new log
I20260925 19:09:44.808450 31941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } }
I20260925 19:09:44.808580 31941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:44.808625 31941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc5c21658ad44a1b3a33d60b6c6e022, State: Initialized, Role: FOLLOWER
I20260925 19:09:44.808785 31941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } }
I20260925 19:09:44.808847 31941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:44.808889 31941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:44.808940 31941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:44.809780 31941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } }
I20260925 19:09:44.809890 31941 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc5c21658ad44a1b3a33d60b6c6e022; no voters:
I20260925 19:09:44.810074 31941 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:44.810137 31944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:44.810328 31944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc5c21658ad44a1b3a33d60b6c6e022, State: Running, Role: LEADER
I20260925 19:09:44.810372 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31873
I20260925 19:09:44.810418 31944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } }
I20260925 19:09:44.810463 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:44.810629 28345 external_mini_cluster.cc:1428] Setting key 06370d858465fcddf70a5b631166ef26
I20260925 19:09:44.810770 31945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } } }
I20260925 19:09:44.810786 31946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc5c21658ad44a1b3a33d60b6c6e022. Latest consensus state: current_term: 1 leader_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc5c21658ad44a1b3a33d60b6c6e022" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43561 } } }
I20260925 19:09:44.810945 31946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:44.810889 31945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:44.811146 31941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:44.811579 31948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:44.812140 31948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:44.813570 31948 catalog_manager.cc:1347] Generated new cluster ID: fe3c17f3b3bd457a8a44f8e6a46ec78c
I20260925 19:09:44.813623 31948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:44 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:44.821691 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:44.811220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.27.174.126:43561)
0925 19:09:44.811378 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:44.811386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:44.811465 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:09:44.811643 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:44.811652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:44.811718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:44.811891 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:44.812899 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.814029 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.815112 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.815792 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.815909 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:44.818190 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:44.818194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:44.818194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:44.818205 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:44.820241 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:44.820593 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:44.820596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:44.820596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:44.820688 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:44.820894 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:44.820895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:44.820896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:44.821129 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:44.821281 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:44.821394 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:09:44.821529 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260925 19:09:44.823760 31948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:44.824445 31948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:44.827986 31948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc5c21658ad44a1b3a33d60b6c6e022: Generated new TSK 0
I20260925 19:09:44.828143 31948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:44.854590 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:44.933699 31967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:44.933938 31967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:44.933974 31967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:44.935140 31967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:44.935205 31967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:44.935249 31967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:44.936744 31967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:44.936993 31967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:44.937214 31967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:44.942241 31973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:44.942246 31975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:44.942276 31972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:44.942385 31967 server_base.cc:1034] running on GCE node
I20260925 19:09:44.942678 31967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:44.942925 31967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:44.944099 31967 hybrid_clock.cc:648] HybridClock initialized: now 1790363384944091 us; error 41 us; skew 500 ppm
Sep 25 19:09:44 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363384, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:44.946583 31967 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:44.947019 31967 webserver.cc:460] Webserver started at http://127.27.174.65:41583/ using document root <none> and password file <none>
I20260925 19:09:44.947196 31967 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:44.947234 31967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:44.947322 31967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:44.948338 31967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "ec39324403184564abbbe4e5e685ab68"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "8fce14d7c1b9d070f01d705805a44e06"
server_key_iv: "06ef44d2323219eefb7be742cb9106eb"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.948629 31967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "ec39324403184564abbbe4e5e685ab68"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "8fce14d7c1b9d070f01d705805a44e06"
server_key_iv: "06ef44d2323219eefb7be742cb9106eb"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.949921 31967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:44.950652 31982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:44.950819 31967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:44.950913 31967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "ec39324403184564abbbe4e5e685ab68"
format_stamp: "Formatted at 2026-09-25 19:09:44 on dist-test-slave-fcdn"
server_key: "8fce14d7c1b9d070f01d705805a44e06"
server_key_iv: "06ef44d2323219eefb7be742cb9106eb"
server_key_version: "encryptionkey@0"
I20260925 19:09:44.950991 31967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:44.964462 31967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:44.966643 31967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:44.966799 31967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:44.967005 31967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:44.967283 31967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:44.967332 31967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:44.967370 31967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:44.967396 31967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:44.973361 31967 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41763
I20260925 19:09:44.973405 32095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41763 every 8 connection(s)
I20260925 19:09:44.973713 31967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:44.979117 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 31967
I20260925 19:09:44.979235 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:44.979497 28345 external_mini_cluster.cc:1428] Setting key a5e43efdeb93fa5ada375a722f8e642c
Sep 25 19:09:44 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363384, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:44.983520 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:44.974416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:51795 (local address 127.27.174.126:43561)
0925 19:09:44.974499 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:44.974501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:44.974691 (+ 190us) server_negotiation.cc:407] Connection header received
0925 19:09:44.975663 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:44.975665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:44.975681 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:44.975731 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:44.976388 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.976942 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.977896 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.978129 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.978185 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:44.980590 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:44.980592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:44.980592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:44.980600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:44.982168 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:44.982559 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:44.982560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:44.982560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:44.982623 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:44.982882 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:44.982883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:44.982884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:44.982997 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:44.983070 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:44.983344 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 19:09:44.983432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:09:44.983845 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:44.974388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43561 (local address 127.27.174.65:51795)
0925 19:09:44.974554 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:44.974607 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:44.975528 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:44.975811 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:44.975815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:44.976063 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:44.976235 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:44.976238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.977024 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.977025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:44.977816 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:44.977818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:44.978197 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:44.978197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:44.978301 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:44.978761 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:44.978769 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:44.980491 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:44.982256 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:44.982261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:44.982263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:44.982485 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:44.982698 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:44.982698 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:44.982699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:44.982796 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:44.983071 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:44.983078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:44.983206 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:09:44.983309 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 19:09:44.983465 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260925 19:09:44.984508 32096 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43561
I20260925 19:09:44.984628 32096 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:44.984797 32096 heartbeater.cc:511] Master 127.27.174.126:43561 requested a full tablet report, sending...
I20260925 19:09:44.985423 31905 ts_manager.cc:194] Registered new tserver with Master: ec39324403184564abbbe4e5e685ab68 (127.27.174.65:41763)
I20260925 19:09:44.986042 31905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:51795
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:45.006166 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:45.088112 32101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:45.088296 32101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:45.088318 32101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:45.089354 32101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:45.089385 32101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:45.089411 32101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:45.090773 32101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:45.090975 32101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:45.091128 32101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:45.095844 32106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:45.095955 32101 server_base.cc:1034] running on GCE node
W20260925 19:09:45.095839 32109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:45.095839 32107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:45.096254 32101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:45.096484 32101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:45.097672 32101 hybrid_clock.cc:648] HybridClock initialized: now 1790363385097659 us; error 38 us; skew 500 ppm
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:45.100099 32101 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:45.100574 32101 webserver.cc:460] Webserver started at http://127.27.174.66:34207/ using document root <none> and password file <none>
I20260925 19:09:45.100803 32101 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:45.100871 32101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:45.101019 32101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:45.102021 32101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "269b6cc53e0446c1af0228fe95e90aa3"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "b61767dc954b8d48740bbfc9cf789365"
server_key_iv: "2c93f2c6d03288dac5e8a223caeca46e"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.102386 32101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "269b6cc53e0446c1af0228fe95e90aa3"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "b61767dc954b8d48740bbfc9cf789365"
server_key_iv: "2c93f2c6d03288dac5e8a223caeca46e"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.103729 32101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:09:45.104417 32116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.104592 32101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:45.104686 32101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "269b6cc53e0446c1af0228fe95e90aa3"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "b61767dc954b8d48740bbfc9cf789365"
server_key_iv: "2c93f2c6d03288dac5e8a223caeca46e"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.104773 32101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:45.117650 32101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:45.119575 32101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:45.119758 32101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:45.119982 32101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:45.120322 32101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:45.120376 32101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.120416 32101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:45.120442 32101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.126080 32101 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44111
I20260925 19:09:45.126132 32229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44111 every 8 connection(s)
I20260925 19:09:45.126459 32101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:45.130434 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32101
I20260925 19:09:45.130544 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:45.130717 28345 external_mini_cluster.cc:1428] Setting key 9c3d4df6bf61a7625e2195e3e552b94f
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:45.136425 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.127206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:56943 (local address 127.27.174.126:43561)
0925 19:09:45.127318 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:45.127321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:45.127468 (+ 147us) server_negotiation.cc:407] Connection header received
0925 19:09:45.128450 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:45.128452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:45.128465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:45.128515 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:45.129119 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.129624 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.130992 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.131181 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.131238 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.133595 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:45.133597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:45.133597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:45.133605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:45.134992 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.135417 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.135418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.135418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.135479 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.135780 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.135781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.135782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.135900 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:45.135978 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:45.136246 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 19:09:45.136336 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 19:09:45.136618 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.127122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43561 (local address 127.27.174.66:56943)
0925 19:09:45.127337 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:45.127382 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:45.128326 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:45.128587 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:45.128591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:45.128844 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:45.129024 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:45.129027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.129739 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.129740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:45.130816 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:45.130819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.131258 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.131259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:45.131383 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.131954 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:45.131964 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:45.133500 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:45.135090 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:45.135096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:45.135099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:45.135329 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:45.135546 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:45.135547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:45.135547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:45.135702 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:45.135968 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:45.135974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:45.136113 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 19:09:45.136215 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:09:45.136377 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260925 19:09:45.137105 32230 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43561
I20260925 19:09:45.137198 32230 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:45.137403 32230 heartbeater.cc:511] Master 127.27.174.126:43561 requested a full tablet report, sending...
I20260925 19:09:45.137781 31905 ts_manager.cc:194] Registered new tserver with Master: 269b6cc53e0446c1af0228fe95e90aa3 (127.27.174.66:44111)
I20260925 19:09:45.138316 31905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:56943
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:45.157920 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:45.237920 32235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:45.238128 32235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:45.238152 32235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:45.239236 32235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:45.239269 32235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:45.239295 32235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:45.240731 32235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:45.240907 32235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:45.241073 32235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:45.245952 32243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:45.245952 32240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:45.245999 32235 server_base.cc:1034] running on GCE node
W20260925 19:09:45.245956 32241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:45.246351 32235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:45.246599 32235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:45.247764 32235 hybrid_clock.cc:648] HybridClock initialized: now 1790363385247742 us; error 33 us; skew 500 ppm
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:45.250233 32235 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:45.250625 32235 webserver.cc:460] Webserver started at http://127.27.174.67:37145/ using document root <none> and password file <none>
I20260925 19:09:45.250800 32235 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:45.250840 32235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:45.250926 32235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:45.251978 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "1a3829e205854b2c86bddfd7640312d0"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "457daa349921d70c4af7bee36b9a038b"
server_key_iv: "937af7556b171ab191b4b3d2da764b29"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.252285 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "1a3829e205854b2c86bddfd7640312d0"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "457daa349921d70c4af7bee36b9a038b"
server_key_iv: "937af7556b171ab191b4b3d2da764b29"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.253549 32235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:45.254266 32250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.254428 32235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:45.254504 32235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "1a3829e205854b2c86bddfd7640312d0"
format_stamp: "Formatted at 2026-09-25 19:09:45 on dist-test-slave-fcdn"
server_key: "457daa349921d70c4af7bee36b9a038b"
server_key_iv: "937af7556b171ab191b4b3d2da764b29"
server_key_version: "encryptionkey@0"
I20260925 19:09:45.254603 32235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:45.268813 32235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:45.270702 32235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:45.270850 32235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:45.271075 32235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:45.271395 32235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:45.271448 32235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.271487 32235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:45.271515 32235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:45.277074 32235 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:45981
I20260925 19:09:45.277130 32363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:45981 every 8 connection(s)
I20260925 19:09:45.277465 32235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:45.282611 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32235
I20260925 19:09:45.282720 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:45.282903 28345 external_mini_cluster.cc:1428] Setting key 6f57801eb30bfd2660dd94c941b029a1
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:45.286741 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.278227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44979 (local address 127.27.174.126:43561)
0925 19:09:45.278292 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:45.278294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:45.278505 (+ 211us) server_negotiation.cc:407] Connection header received
0925 19:09:45.279385 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:45.279386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:45.279395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:45.279452 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:45.280107 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.280585 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.281531 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.281708 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.281759 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.284181 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:45.284182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:45.284183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:45.284190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:45.285408 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.285797 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.285798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.285799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.285855 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.286112 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.286113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.286113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.286231 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:45.286310 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:45.286571 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 19:09:45.286653 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:09:45.286924 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.278198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43561 (local address 127.27.174.67:44979)
0925 19:09:45.278376 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:45.278417 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:45.279297 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:45.279506 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:45.279509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:45.279838 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:45.280027 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:45.280032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.280660 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.280660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:45.281466 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:45.281468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.281778 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.281779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:45.281875 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.282397 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:45.282406 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:45.284090 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:45.285496 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:45.285502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:45.285504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:45.285717 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:45.285929 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:45.285930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:45.285930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:45.286045 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:45.286300 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:45.286307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:45.286442 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:09:45.286550 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:09:45.286702 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260925 19:09:45.287395 32364 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43561
I20260925 19:09:45.287487 32364 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:45.287720 32364 heartbeater.cc:511] Master 127.27.174.126:43561 requested a full tablet report, sending...
I20260925 19:09:45.288079 31905 ts_manager.cc:194] Registered new tserver with Master: 1a3829e205854b2c86bddfd7640312d0 (127.27.174.67:45981)
I20260925 19:09:45.288448 31905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:44979
I20260925 19:09:45.294806 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:45.317204 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.309123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.27.174.65:41763)
0925 19:09:45.309297 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:45.309302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:45.309314 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:45.309342 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:45.309343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:45.309356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:45.309432 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:45.310053 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.310876 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.311752 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.312202 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.312290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.314548 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:45.314556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:45.314557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:45.314564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:45.316007 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.316370 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.316373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.316374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.316444 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.316653 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.316654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.316655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.316783 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:45.316931 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:45.316997 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:09:45.317071 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[31860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:45.324919 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:45.318477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.27.174.126:43561)
0925 19:09:45.318518 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:45.318520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:45.318530 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:45.318581 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:45.318582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:45.318590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:45.318699 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:45.319089 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.319587 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.320307 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:45.320494 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:45.320544 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:45.322652 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:45.322653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:45.322654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:45.322660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:45.323960 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.324280 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.324281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.324281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.324342 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:45.324496 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:45.324497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:45.324497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:45.324624 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:45.324723 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:45.324808 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 19:09:45.324860 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:09:45.325165 31905 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60490
I20260925 19:09:45.325618 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31967
I20260925 19:09:45.330670 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32101
I20260925 19:09:45.335522 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32235
I20260925 19:09:45.339843 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 31873
2026-09-25T19:09:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[32376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[32376](info): set up 2 sockets
Sep 25 19:09:45 dist-test-slave-fcdn krb5kdc[32376](info): commencing operation
krb5kdc: starting...
W20260925 19:09:46.371479 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:46.486603 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44139
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:46.565205 32389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:46.565404 32389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:46.565425 32389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:46.566557 32389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:46.566599 32389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:46.566612 32389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:46.566625 32389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:46.566674 32389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:46.568137 32389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:46.568379 32389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:46.568602 32389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:46.573072 32395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:46.573083 32394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:46.573138 32397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:46.573369 32389 server_base.cc:1034] running on GCE node
I20260925 19:09:46.573591 32389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:46.573823 32389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:46.574975 32389 hybrid_clock.cc:648] HybridClock initialized: now 1790363386574957 us; error 33 us; skew 500 ppm
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:46.577360 32389 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:46.577625 32389 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:46.577665 32389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:46.577757 32389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:46.578685 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "ce61fcedfd3d47b3a1d540f618ad8d61"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "08af4ae43cf8390239dab384d05c2e13"
server_key_iv: "75fa4defb2cdeccf2e891f621bbd1152"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.578984 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ce61fcedfd3d47b3a1d540f618ad8d61"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "08af4ae43cf8390239dab384d05c2e13"
server_key_iv: "75fa4defb2cdeccf2e891f621bbd1152"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.580307 32389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:46.581025 32403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.581185 32389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:46.581290 32389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "ce61fcedfd3d47b3a1d540f618ad8d61"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "08af4ae43cf8390239dab384d05c2e13"
server_key_iv: "75fa4defb2cdeccf2e891f621bbd1152"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.581360 32389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:46.591966 32389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:46.594149 32389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:46.594316 32389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:46.598130 32389 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44139
I20260925 19:09:46.598158 32455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44139 every 8 connection(s)
I20260925 19:09:46.598430 32389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:46.599313 32456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:46.600780 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32389
I20260925 19:09:46.600890 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:46.601074 28345 external_mini_cluster.cc:1428] Setting key 228560ce16d2132813f099aefa760439
I20260925 19:09:46.602084 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: Bootstrap starting.
I20260925 19:09:46.602810 32456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:46.603171 32456 log.cc:824] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:46.603888 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: No bootstrap required, opened a new log
I20260925 19:09:46.606056 32456 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } }
I20260925 19:09:46.606225 32456 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:46.606251 32456 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce61fcedfd3d47b3a1d540f618ad8d61, State: Initialized, Role: FOLLOWER
I20260925 19:09:46.606400 32456 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } }
I20260925 19:09:46.606490 32456 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:46.606525 32456 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:46.606572 32456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:46.607070 32456 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } }
I20260925 19:09:46.607206 32456 leader_election.cc:302] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce61fcedfd3d47b3a1d540f618ad8d61; no voters:
I20260925 19:09:46.607378 32456 leader_election.cc:288] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:46.607445 32461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:46.607589 32461 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [term 1 LEADER]: Becoming Leader. State: Replica: ce61fcedfd3d47b3a1d540f618ad8d61, State: Running, Role: LEADER
I20260925 19:09:46.607640 32456 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:46.607698 32461 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } }
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:46.607996 32461 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce61fcedfd3d47b3a1d540f618ad8d61. Latest consensus state: current_term: 1 leader_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } } }
I20260925 19:09:46.608080 32461 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:46.608006 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce61fcedfd3d47b3a1d540f618ad8d61" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44139 } } }
I20260925 19:09:46.608316 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:46.609750 32477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:46.609810 32477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:46.610498 32468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:46.610674 32468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:46.612073 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:46.601585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.27.174.126:44139)
0925 19:09:46.601750 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:46.601758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:46.601782 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:09:46.602199 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:46.602209 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:46.602263 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:46.602373 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:46.603185 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.604329 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.605430 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.605807 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.605881 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:46.608608 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:46.608612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:46.608612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:46.608623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:46.610381 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.610801 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.610805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.610805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.610906 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.611143 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.611144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.611145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.611420 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:46.611575 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:46.611725 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:09:46.611864 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 19:09:46.612284 32468 catalog_manager.cc:1347] Generated new cluster ID: 2e52cc1f4020471e92d802e357ae7855
I20260925 19:09:46.612327 32468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:46.623865 32468 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:46.624490 32468 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:46.632077 32468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: Generated new TSK 0
I20260925 19:09:46.632251 32468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:46.658425 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:46.737847 32482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:46.738080 32482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:46.738118 32482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:46.739279 32482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:46.739342 32482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:46.739387 32482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:46.740896 32482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:46.741154 32482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:46.741379 32482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:46.746264 32487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:46.746248 32488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:46.746274 32482 server_base.cc:1034] running on GCE node
W20260925 19:09:46.746248 32490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:46.746762 32482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:46.746994 32482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:46.748135 32482 hybrid_clock.cc:648] HybridClock initialized: now 1790363386748113 us; error 34 us; skew 500 ppm
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:46.750597 32482 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:46.750906 32482 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:46.750950 32482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:46.751107 32482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:46.752179 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "83423c2877684f1595e02f563a3d0906"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "243de42a83170433b1cc75687b5f46c2"
server_key_iv: "e9ba3374b48f632926d808da96a59019"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.752529 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "83423c2877684f1595e02f563a3d0906"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "243de42a83170433b1cc75687b5f46c2"
server_key_iv: "e9ba3374b48f632926d808da96a59019"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.753751 32482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:46.754462 32496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.754643 32482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:46.754714 32482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "83423c2877684f1595e02f563a3d0906"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "243de42a83170433b1cc75687b5f46c2"
server_key_iv: "e9ba3374b48f632926d808da96a59019"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.754802 32482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:46.764576 32482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:46.766775 32482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:46.766886 32482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:46.767040 32482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:46.767340 32482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:46.767375 32482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.767396 32482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:46.767443 32482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.773109 32482 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39647
I20260925 19:09:46.773205 32609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39647 every 8 connection(s)
I20260925 19:09:46.773437 32482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:46.782042 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:46.774133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47367 (local address 127.27.174.126:44139)
0925 19:09:46.774178 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:46.774180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:46.774383 (+ 203us) server_negotiation.cc:407] Connection header received
0925 19:09:46.774837 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:46.774837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:46.774846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:46.774907 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:46.775862 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.776356 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.777291 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.777486 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.777544 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:46.779616 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:46.779618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:46.779619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:46.779626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:46.780944 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.781290 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.781290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.781291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.781342 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.781540 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.781540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.781541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.781649 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:46.781700 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:46.781921 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 19:09:46.781980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:46.782182 32612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:46.774078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44139 (local address 127.27.174.65:47367)
0925 19:09:46.774246 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:46.774285 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:46.774754 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:46.774967 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:46.774970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:46.775214 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:46.775759 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:46.775765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.776435 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.776436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:46.777202 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:46.777204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.777560 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.777561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:46.777653 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:46.778139 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:46.778148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:46.779514 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:46.781014 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:46.781019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:46.781021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:46.781225 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:46.781397 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:46.781397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:46.781397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:46.781487 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:46.781710 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:46.781714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:46.781814 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 19:09:46.781894 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:09:46.782002 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260925 19:09:46.782670 32610 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44139
I20260925 19:09:46.782778 32610 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:46.782919 32610 heartbeater.cc:511] Master 127.27.174.126:44139 requested a full tablet report, sending...
I20260925 19:09:46.783209 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32482
I20260925 19:09:46.783317 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:46.783483 32420 ts_manager.cc:194] Registered new tserver with Master: 83423c2877684f1595e02f563a3d0906 (127.27.174.65:39647)
I20260925 19:09:46.783509 28345 external_mini_cluster.cc:1428] Setting key 0e17ce00a93d2e199be65f4251756ce8
I20260925 19:09:46.784536 32420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:47367
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:46.808506 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:46.888798 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:46.888989 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:46.889012 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:46.890156 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:46.890201 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:46.890228 32615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:46.891706 32615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:46.891928 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:46.892144 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:46.896914 32620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:46.896976 32615 server_base.cc:1034] running on GCE node
W20260925 19:09:46.896924 32623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:46.896947 32621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:46.897333 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:46.897513 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:46.898684 32615 hybrid_clock.cc:648] HybridClock initialized: now 1790363386898654 us; error 38 us; skew 500 ppm
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:46.901029 32615 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:46.901347 32615 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:46.901415 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:46.901556 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:46.902581 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "0d4588dc169d4fadb238fe580c2397ac"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "fdc373139a1d9dadcf44dc56a63839cf"
server_key_iv: "793ee1b5b81a603acf40649295b18ac4"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.902917 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "0d4588dc169d4fadb238fe580c2397ac"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "fdc373139a1d9dadcf44dc56a63839cf"
server_key_iv: "793ee1b5b81a603acf40649295b18ac4"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.904225 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:46.904892 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.905047 32615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:46.905143 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "0d4588dc169d4fadb238fe580c2397ac"
format_stamp: "Formatted at 2026-09-25 19:09:46 on dist-test-slave-fcdn"
server_key: "fdc373139a1d9dadcf44dc56a63839cf"
server_key_iv: "793ee1b5b81a603acf40649295b18ac4"
server_key_version: "encryptionkey@0"
I20260925 19:09:46.905225 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:46.917402 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:46.919186 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:46.919339 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:46.919552 32615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:46.919857 32615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:46.919924 32615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.919962 32615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:46.919989 32615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:46.925711 32615 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:35401
I20260925 19:09:46.925776 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:35401 every 8 connection(s)
I20260925 19:09:46.926033 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:09:46 dist-test-slave-fcdn krb5kdc[32376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:46.932775 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32615
I20260925 19:09:46.932850 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:46.933039 28345 external_mini_cluster.cc:1428] Setting key d7e95939b037b787e56ef67c8c1213e5
I20260925 19:09:46.935222 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:46.926717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:35251 (local address 127.27.174.126:44139)
0925 19:09:46.926762 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:46.926764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:46.926974 (+ 210us) server_negotiation.cc:407] Connection header received
0925 19:09:46.927402 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:46.927403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:46.927412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:46.927461 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:46.928376 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.928838 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.929777 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.929946 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.929997 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:46.932133 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:46.932134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:46.932134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:46.932141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:46.933798 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.934279 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.934279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.934280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.934330 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:46.934617 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:46.934617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:46.934617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:46.934709 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:46.934758 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:46.935078 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 19:09:46.935141 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:46.935495 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:46.926712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44139 (local address 127.27.174.66:35251)
0925 19:09:46.926861 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:46.926893 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:46.927328 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:46.927509 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:46.927512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:46.927856 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:46.928304 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:46.928309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.928922 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.928922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:46.929706 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:46.929708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:46.930030 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:46.930031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:46.930146 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:46.930621 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:46.930637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:46.932043 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:46.933877 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:46.933883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:46.933885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:46.934202 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:46.934388 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:46.934389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:46.934389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:46.934543 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:46.934772 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:46.934777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:46.934941 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 19:09:46.935058 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 19:09:46.935231 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260925 19:09:46.936012 32743 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44139
I20260925 19:09:46.936116 32743 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:46.936256 32743 heartbeater.cc:511] Master 127.27.174.126:44139 requested a full tablet report, sending...
I20260925 19:09:46.936620 32420 ts_manager.cc:194] Registered new tserver with Master: 0d4588dc169d4fadb238fe580c2397ac (127.27.174.66:35401)
I20260925 19:09:46.937011 32420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:35251
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:46.958324 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:47.039178 32748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:47.039371 32748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:47.039393 32748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:47.040578 32748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:47.040622 32748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:47.040649 32748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:47.042145 32748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:47.042375 32748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:47.042604 32748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:47.047394 32754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.047394 32756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.047482 32748 server_base.cc:1034] running on GCE node
W20260925 19:09:47.047443 32753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.047850 32748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:47.048079 32748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:47.049223 32748 hybrid_clock.cc:648] HybridClock initialized: now 1790363387049210 us; error 30 us; skew 500 ppm
Sep 25 19:09:47 dist-test-slave-fcdn krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:47.051733 32748 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:47.052085 32748 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:47.052136 32748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:47.052225 32748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:47.053171 32748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "bace820905e24b348abad07e7263f637"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "0f33f5f88d01581a68afee7b3d145777"
server_key_iv: "4c4f5f5b25247e54b54b6272ee386493"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.053462 32748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "bace820905e24b348abad07e7263f637"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "0f33f5f88d01581a68afee7b3d145777"
server_key_iv: "4c4f5f5b25247e54b54b6272ee386493"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.054708 32748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:47.055413 32762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.055660 32748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:47.055773 32748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "bace820905e24b348abad07e7263f637"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "0f33f5f88d01581a68afee7b3d145777"
server_key_iv: "4c4f5f5b25247e54b54b6272ee386493"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.055856 32748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:47.070211 32748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:47.072067 32748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:47.072234 32748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:47.072454 32748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:47.072765 32748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:47.072821 32748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.072860 32748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:47.072887 32748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.078794 32748 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42445
I20260925 19:09:47.078836 407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42445 every 8 connection(s)
I20260925 19:09:47.079098 32748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:47.083009 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 32748
I20260925 19:09:47.083113 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:47.083345 28345 external_mini_cluster.cc:1428] Setting key 2519dfd2a72b72304285c451173e7d5d
Sep 25 19:09:47 dist-test-slave-fcdn krb5kdc[32376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:47.088876 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.079794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:32999 (local address 127.27.174.126:44139)
0925 19:09:47.079852 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.079854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.080046 (+ 192us) server_negotiation.cc:407] Connection header received
0925 19:09:47.080568 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.080569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.080582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.080643 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.081632 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.082120 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.083272 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.083549 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.083685 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.086273 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.086274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.086274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:47.086280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.087773 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:47.088142 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:47.088142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:47.088143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:47.088190 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:47.088391 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:47.088391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:47.088391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:47.088485 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.088532 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.088751 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.088814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:09:47.089113 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.079775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44139 (local address 127.27.174.67:32999)
0925 19:09:47.079919 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.079966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.080461 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.080715 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.080719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.080984 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.081532 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.081538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.082213 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.082214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.083187 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.083189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.083693 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.083694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.083831 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.084369 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:47.084377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.086165 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.087846 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:47.087851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:47.087853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:47.088070 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:47.088246 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:47.088247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:47.088247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:47.088338 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:47.088538 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.088542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.088652 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.088731 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.088845 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260925 19:09:47.089581 408 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44139
I20260925 19:09:47.089659 408 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:47.089764 408 heartbeater.cc:511] Master 127.27.174.126:44139 requested a full tablet report, sending...
I20260925 19:09:47.090067 32420 ts_manager.cc:194] Registered new tserver with Master: bace820905e24b348abad07e7263f637 (127.27.174.67:42445)
I20260925 19:09:47.090440 32420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:32999
I20260925 19:09:47.094827 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:09:47.101543 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.096243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.27.174.126:44139)
0925 19:09:47.096299 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.096300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.096429 (+ 129us) server_negotiation.cc:407] Connection header received
0925 19:09:47.096465 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.096465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.096473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.096523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.097021 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.097472 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.098138 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.098322 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.098367 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.099346 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.099347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.099347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:47.099353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.100707 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:47.100984 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:47.100985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:47.100985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:47.101024 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:47.101165 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:47.101165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:47.101165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:47.101287 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.101356 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.101435 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.101486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:09:47.102960 32420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:47.104624 32420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:47.111123 417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.106708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39647 (local address 127.0.0.1:35606)
0925 19:09:47.107064 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.107075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.107136 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.107355 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.107357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.107360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:47.107425 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.107427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.108695 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.108696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.110087 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.110090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110908 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.110968 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110971 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.111021 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.111053 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":40,"thread_start_us":53,"threads_started":1}
I20260925 19:09:47.111155 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.106846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.27.174.65:39647)
0925 19:09:47.106990 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.107013 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.107124 (+ 111us) server_negotiation.cc:407] Connection header received
0925 19:09:47.107210 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.107255 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.107273 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.107349 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:47.107510 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.108592 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110170 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110847 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110911 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110956 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.111034 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.111078 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260925 19:09:47.111203 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.106756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.27.174.67:42445)
0925 19:09:47.106889 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.106893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.107086 (+ 193us) server_negotiation.cc:407] Connection header received
0925 19:09:47.107210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.107212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.107228 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.107282 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:47.107524 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.108614 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110033 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110751 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110821 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110879 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.111072 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.111111 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260925 19:09:47.111366 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.106895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:35401 (local address 127.0.0.1:34008)
0925 19:09:47.107314 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.107323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.107394 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.107711 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.107712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.107714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:47.107757 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.107759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.109078 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.109079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.110158 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.110160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110795 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.110869 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110883 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.110950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.111007 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":35,"threads_started":1}
I20260925 19:09:47.111418 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.106702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42445 (local address 127.0.0.1:58892)
0925 19:09:47.106976 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.107035 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.107105 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.107392 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.107393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.107395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:47.107440 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.107442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.108733 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.108734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.109946 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.109949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110915 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.110980 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110984 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.111274 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.111321 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":47,"thread_start_us":53,"threads_started":1}
I20260925 19:09:47.111940 32544 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0a8a9ae0f64972917338b50208d53c (DEFAULT_TABLE table=test-table [id=ab44f377b9c34d9880e69b4a78b6d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.112109 342 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0a8a9ae0f64972917338b50208d53c (DEFAULT_TABLE table=test-table [id=ab44f377b9c34d9880e69b4a78b6d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.112321 32544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0a8a9ae0f64972917338b50208d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.112455 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0a8a9ae0f64972917338b50208d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.115558 422 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: Bootstrap starting.
I20260925 19:09:47.115581 423 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Bootstrap starting.
I20260925 19:09:47.115411 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.107096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.27.174.66:35401)
0925 19:09:47.107279 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.107284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.107471 (+ 187us) server_negotiation.cc:407] Connection header received
0925 19:09:47.107513 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.107516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.107530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.107622 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:47.107827 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.108995 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110247 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.110702 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.110796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.110853 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.111092 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.111134 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260925 19:09:47.116261 422 tablet_bootstrap.cc:654] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.116267 423 tablet_bootstrap.cc:654] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.116922 422 log.cc:824] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.117120 32677 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0a8a9ae0f64972917338b50208d53c (DEFAULT_TABLE table=test-table [id=ab44f377b9c34d9880e69b4a78b6d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.117473 32677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0a8a9ae0f64972917338b50208d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.117820 422 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: No bootstrap required, opened a new log
I20260925 19:09:47.117899 422 ts_tablet_manager.cc:1397] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:47.119403 423 log.cc:824] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.120126 423 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: No bootstrap required, opened a new log
I20260925 19:09:47.120199 423 ts_tablet_manager.cc:1397] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 19:09:47.120431 422 raft_consensus.cc:348] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.120597 422 raft_consensus.cc:374] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.120630 422 raft_consensus.cc:729] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bace820905e24b348abad07e7263f637, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.120781 422 consensus_queue.cc:260] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.121024 422 ts_tablet_manager.cc:1428] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:47.121147 408 heartbeater.cc:503] Master 127.27.174.126:44139 was elected leader, sending a full tablet report...
I20260925 19:09:47.121627 427 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Bootstrap starting.
I20260925 19:09:47.122265 427 tablet_bootstrap.cc:654] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.122278 423 raft_consensus.cc:348] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.122447 423 raft_consensus.cc:374] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.122485 423 raft_consensus.cc:729] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83423c2877684f1595e02f563a3d0906, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.122612 423 consensus_queue.cc:260] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.122834 427 log.cc:824] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.122920 423 ts_tablet_manager.cc:1428] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:09:47.122973 32610 heartbeater.cc:503] Master 127.27.174.126:44139 was elected leader, sending a full tablet report...
I20260925 19:09:47.124254 427 tablet_bootstrap.cc:492] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: No bootstrap required, opened a new log
I20260925 19:09:47.124320 427 ts_tablet_manager.cc:1397] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:09:47.126622 427 raft_consensus.cc:348] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.126796 427 raft_consensus.cc:374] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.126840 427 raft_consensus.cc:729] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4588dc169d4fadb238fe580c2397ac, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.126957 427 consensus_queue.cc:260] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.127211 427 ts_tablet_manager.cc:1428] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:47.127221 32743 heartbeater.cc:503] Master 127.27.174.126:44139 was elected leader, sending a full tablet report...
W20260925 19:09:47.176522 32744 tablet.cc:2369] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:47.266623 426 raft_consensus.cc:482] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:47.266806 426 raft_consensus.cc:504] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.267182 426 leader_election.cc:288] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83423c2877684f1595e02f563a3d0906 (127.27.174.65:39647), 0d4588dc169d4fadb238fe580c2397ac (127.27.174.66:35401)
I20260925 19:09:47.271391 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.267501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40489 (local address 127.27.174.66:35401)
0925 19:09:47.267558 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.267561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.267589 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:09:47.267695 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.267696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.267704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.267792 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:47.268048 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.268818 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.270437 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.271038 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.271102 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.271111 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.271282 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.271323 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:09:47.271391 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.267458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34251 (local address 127.27.174.65:39647)
0925 19:09:47.267503 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.267505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.267762 (+ 257us) server_negotiation.cc:407] Connection header received
0925 19:09:47.267803 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.267805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.267814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.267872 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:47.268048 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.268798 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.270443 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.270987 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.271070 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.271081 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.271283 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.271321 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:47.271396 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.267421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:35401 (local address 127.27.174.67:40489)
0925 19:09:47.267474 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.267487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.267533 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.267807 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.267808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.267809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:47.267900 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.267902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.268921 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.268922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.270322 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.270325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.271111 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.271112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.271199 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.271201 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.271262 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.271308 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":67}
I20260925 19:09:47.271405 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.267422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39647 (local address 127.27.174.67:34251)
0925 19:09:47.267674 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.267683 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.267740 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.267883 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.267883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.267894 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:47.267944 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.267945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.268921 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.268922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.270322 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:47.270325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.271101 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.271103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:47.271192 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.271197 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.271262 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.271307 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":135,"thread_start_us":137,"threads_started":1}
I20260925 19:09:47.271763 32564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0a8a9ae0f64972917338b50208d53c" candidate_uuid: "bace820905e24b348abad07e7263f637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83423c2877684f1595e02f563a3d0906" is_pre_election: true
I20260925 19:09:47.271811 32697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0a8a9ae0f64972917338b50208d53c" candidate_uuid: "bace820905e24b348abad07e7263f637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4588dc169d4fadb238fe580c2397ac" is_pre_election: true
I20260925 19:09:47.271936 32564 raft_consensus.cc:2454] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bace820905e24b348abad07e7263f637 in term 0.
I20260925 19:09:47.271986 32697 raft_consensus.cc:2454] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bace820905e24b348abad07e7263f637 in term 0.
I20260925 19:09:47.272192 32764 leader_election.cc:302] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83423c2877684f1595e02f563a3d0906, bace820905e24b348abad07e7263f637; no voters:
I20260925 19:09:47.272331 426 raft_consensus.cc:2781] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:47.272385 426 raft_consensus.cc:482] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:47.272408 426 raft_consensus.cc:3037] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.273108 426 raft_consensus.cc:504] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.273264 426 leader_election.cc:288] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [CANDIDATE]: Term 1 election: Requested vote from peers 83423c2877684f1595e02f563a3d0906 (127.27.174.65:39647), 0d4588dc169d4fadb238fe580c2397ac (127.27.174.66:35401)
I20260925 19:09:47.273425 32564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0a8a9ae0f64972917338b50208d53c" candidate_uuid: "bace820905e24b348abad07e7263f637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83423c2877684f1595e02f563a3d0906"
I20260925 19:09:47.273473 32697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0a8a9ae0f64972917338b50208d53c" candidate_uuid: "bace820905e24b348abad07e7263f637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4588dc169d4fadb238fe580c2397ac"
I20260925 19:09:47.273502 32564 raft_consensus.cc:3037] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.273562 32697 raft_consensus.cc:3037] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 0 FOLLOWER]: Advancing to term 1
W20260925 19:09:47.274104 32611 tablet.cc:2369] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:47.274322 32564 raft_consensus.cc:2454] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bace820905e24b348abad07e7263f637 in term 1.
I20260925 19:09:47.274426 32697 raft_consensus.cc:2454] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bace820905e24b348abad07e7263f637 in term 1.
I20260925 19:09:47.274478 32764 leader_election.cc:302] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83423c2877684f1595e02f563a3d0906, bace820905e24b348abad07e7263f637; no voters:
I20260925 19:09:47.274562 426 raft_consensus.cc:2781] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:47.274729 426 raft_consensus.cc:686] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 1 LEADER]: Becoming Leader. State: Replica: bace820905e24b348abad07e7263f637, State: Running, Role: LEADER
I20260925 19:09:47.274837 426 consensus_queue.cc:237] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } }
I20260925 19:09:47.275784 32419 catalog_manager.cc:5495] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 reported cstate change: term changed from 0 to 1, leader changed from <none> to bace820905e24b348abad07e7263f637 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "bace820905e24b348abad07e7263f637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bace820905e24b348abad07e7263f637" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:47.291288 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.289095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.27.174.67:42445)
0925 19:09:47.289158 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.289159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.289168 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:47.289233 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.289233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.289250 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.289310 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:47.289428 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.289896 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.290595 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:47.290778 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:47.290827 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:47.290987 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:47.291079 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:47.291139 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.291197 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.291212 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:09:47.293135 32564 raft_consensus.cc:1261] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 1 FOLLOWER]: Refusing update from remote peer bace820905e24b348abad07e7263f637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:47.293149 32697 raft_consensus.cc:1261] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 1 FOLLOWER]: Refusing update from remote peer bace820905e24b348abad07e7263f637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:47.293427 426 consensus_queue.cc:1035] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4588dc169d4fadb238fe580c2397ac" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:47.293546 432 consensus_queue.cc:1035] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83423c2877684f1595e02f563a3d0906" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:47.295881 437 mvcc.cc:205] Tried to move back new op lower bound from 7333328434349928448 to 7333328434278346752. Current Snapshot: MvccSnapshot[applied={T|T < 7333328434349928448}]
I20260925 19:09:47.296394 435 mvcc.cc:205] Tried to move back new op lower bound from 7333328434349928448 to 7333328434278346752. Current Snapshot: MvccSnapshot[applied={T|T < 7333328434349928448 or (T in {7333328434349928448})}]
I20260925 19:09:47.296756 436 mvcc.cc:205] Tried to move back new op lower bound from 7333328434349928448 to 7333328434278346752. Current Snapshot: MvccSnapshot[applied={T|T < 7333328434349928448}]
I20260925 19:09:47.297968 32420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:47.298063 32420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:47.298775 32420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce61fcedfd3d47b3a1d540f618ad8d61: Sending DeleteTablet for 3 replicas of tablet 7f0a8a9ae0f64972917338b50208d53c
I20260925 19:09:47.299116 32544 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0a8a9ae0f64972917338b50208d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35606
I20260925 19:09:47.299187 32677 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0a8a9ae0f64972917338b50208d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:34008
I20260925 19:09:47.299296 342 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0a8a9ae0f64972917338b50208d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:58892
I20260925 19:09:47.299423 445 tablet_replica.cc:330] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: stopping tablet replica
I20260925 19:09:47.299515 445 raft_consensus.cc:2229] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:47.299515 444 tablet_replica.cc:330] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906: stopping tablet replica
I20260925 19:09:47.299631 444 raft_consensus.cc:2229] T 7f0a8a9ae0f64972917338b50208d53c P 83423c2877684f1595e02f563a3d0906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:47.299639 445 raft_consensus.cc:2258] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:47.299649 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32482
I20260925 19:09:47.300792 445 ts_tablet_manager.cc:1905] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:47.302326 445 ts_tablet_manager.cc:1918] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:47.302397 445 log.cc:1196] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/7f0a8a9ae0f64972917338b50208d53c
I20260925 19:09:47.302659 445 ts_tablet_manager.cc:1939] T 7f0a8a9ae0f64972917338b50208d53c P 0d4588dc169d4fadb238fe580c2397ac: Deleting consensus metadata
I20260925 19:09:47.303089 32407 catalog_manager.cc:4841] TS 0d4588dc169d4fadb238fe580c2397ac (127.27.174.66:35401): tablet 7f0a8a9ae0f64972917338b50208d53c (table test-table [id=ab44f377b9c34d9880e69b4a78b6d1eb]) successfully deleted
W20260925 19:09:47.304812 32405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:39647 (error 108)
I20260925 19:09:47.305154 446 tablet_replica.cc:330] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637: stopping tablet replica
I20260925 19:09:47.305263 446 raft_consensus.cc:2229] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:47.305292 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32615
I20260925 19:09:47.305310 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.305185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39647 (local address 127.0.0.1:35612)
0925 19:09:47.305239 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.305281 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:09:47.305404 32405 catalog_manager.cc:4584] TS 83423c2877684f1595e02f563a3d0906 (127.27.174.65:39647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f0a8a9ae0f64972917338b50208d53c: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39647: connect: Connection refused (error 111)
I20260925 19:09:47.305356 446 raft_consensus.cc:2258] T 7f0a8a9ae0f64972917338b50208d53c P bace820905e24b348abad07e7263f637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:47.310014 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32748
I20260925 19:09:47.314433 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 32389
I20260925 19:09:47.314579 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.314451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42445 (local address 127.0.0.1:58916)
0925 19:09:47.314513 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.314544 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.67:42445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
2026-09-25T19:09:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1974 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T19:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:47Z Disabled control of system clock
I20260925 19:09:47.341073 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41617
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:09:47.420485 451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:47.420678 451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:47.420698 451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:47.421763 451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:47.421794 451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:47.421806 451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:47.421819 451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:47.421831 451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:47.423063 451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:47.423254 451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:47.423493 451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:47.428066 456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.428171 457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.428068 459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.428432 451 server_base.cc:1034] running on GCE node
I20260925 19:09:47.428682 451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:47.428973 451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:47.430156 451 hybrid_clock.cc:648] HybridClock initialized: now 1790363387430127 us; error 38 us; skew 500 ppm
I20260925 19:09:47.431414 451 webserver.cc:460] Webserver started at http://127.27.174.126:36463/ using document root <none> and password file <none>
I20260925 19:09:47.431739 451 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:47.431819 451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:47.431965 451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:47.433187 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "823ddb6725084ccda53b92d8b4956e3f"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6c7893e74cf7e8d0938da22f6cfcbd8e"
server_key_iv: "634fadc7dcc2fc91d9f195a03c812bf8"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.433591 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "823ddb6725084ccda53b92d8b4956e3f"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6c7893e74cf7e8d0938da22f6cfcbd8e"
server_key_iv: "634fadc7dcc2fc91d9f195a03c812bf8"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.434962 451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:47.435745 465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.435918 451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:47.436023 451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "823ddb6725084ccda53b92d8b4956e3f"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6c7893e74cf7e8d0938da22f6cfcbd8e"
server_key_iv: "634fadc7dcc2fc91d9f195a03c812bf8"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.436110 451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:47.439576 451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:47.439882 451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:47.440037 451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:47.444057 451 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41617
I20260925 19:09:47.444097 517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41617 every 8 connection(s)
I20260925 19:09:47.444425 451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:47.444927 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 451
I20260925 19:09:47.445044 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:47.445093 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.445304 28345 external_mini_cluster.cc:1428] Setting key 4652b9cd66ddc2fab9a7880546d697a4
I20260925 19:09:47.448094 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.445830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.27.174.126:41617)
0925 19:09:47.446059 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.446068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.446109 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:09:47.446776 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.446785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.446837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.446986 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.447688 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.447691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.447692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.447696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.447739 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.447753 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.447831 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.447870 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.447923 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 19:09:47.448961 518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: Bootstrap starting.
I20260925 19:09:47.449518 518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.449787 518 log.cc:824] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.450415 518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: No bootstrap required, opened a new log
I20260925 19:09:47.451803 518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } }
I20260925 19:09:47.451923 518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.451946 518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823ddb6725084ccda53b92d8b4956e3f, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.452106 518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } }
I20260925 19:09:47.452188 518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:47.452270 518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:47.452322 518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.452824 518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } }
I20260925 19:09:47.452930 518 leader_election.cc:302] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 823ddb6725084ccda53b92d8b4956e3f; no voters:
I20260925 19:09:47.453099 518 leader_election.cc:288] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:47.453349 518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:47.453388 523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:47.453575 523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [term 1 LEADER]: Becoming Leader. State: Replica: 823ddb6725084ccda53b92d8b4956e3f, State: Running, Role: LEADER
I20260925 19:09:47.453679 523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } }
I20260925 19:09:47.454241 525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "823ddb6725084ccda53b92d8b4956e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } } }
I20260925 19:09:47.454339 525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:47.454469 524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 823ddb6725084ccda53b92d8b4956e3f. Latest consensus state: current_term: 1 leader_uuid: "823ddb6725084ccda53b92d8b4956e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ddb6725084ccda53b92d8b4956e3f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41617 } } }
I20260925 19:09:47.454523 524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:47.454793 532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:47.455272 532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:47.456804 532 catalog_manager.cc:1347] Generated new cluster ID: 279bf4aecba44beeb1b3e70eca2dc222
I20260925 19:09:47.456861 532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:47.462333 532 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:47.462476 532 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:47.471063 532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: Generated new TSK 0
I20260925 19:09:47.471228 532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:47.473906 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--builtin_ntp_servers=127.27.174.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:09:47.553041 542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:47.553333 542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:47.553383 542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:47.554742 542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:47.554780 542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:47.554809 542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:47.556205 542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:47.556443 542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:47.556737 542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:47.562404 548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.562415 550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.562531 542 server_base.cc:1034] running on GCE node
W20260925 19:09:47.562428 547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.562811 542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:47.563043 542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:47.564195 542 hybrid_clock.cc:648] HybridClock initialized: now 1790363387564184 us; error 30 us; skew 500 ppm
I20260925 19:09:47.565366 542 webserver.cc:460] Webserver started at http://127.27.174.65:39633/ using document root <none> and password file <none>
I20260925 19:09:47.565604 542 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:47.565675 542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:47.565832 542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:47.566843 542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "011897d5829b4939bc6be6ab1b23805b"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6cf0a28cdb9585394771579def2e10f8"
server_key_iv: "cd06015c2a860c4c48c182e3a92964e1"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.567193 542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "011897d5829b4939bc6be6ab1b23805b"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6cf0a28cdb9585394771579def2e10f8"
server_key_iv: "cd06015c2a860c4c48c182e3a92964e1"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.568668 542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:47.569432 556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.569598 542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:47.569692 542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "011897d5829b4939bc6be6ab1b23805b"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "6cf0a28cdb9585394771579def2e10f8"
server_key_iv: "cd06015c2a860c4c48c182e3a92964e1"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.569766 542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:47.572981 542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:47.573225 542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:47.573366 542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:47.573565 542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:47.573861 542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:47.573912 542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.573951 542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:47.573976 542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.579761 542 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37173
I20260925 19:09:47.579806 669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37173 every 8 connection(s)
I20260925 19:09:47.580106 542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:47.583089 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.580801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35055 (local address 127.27.174.126:41617)
0925 19:09:47.580844 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.580846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.581090 (+ 244us) server_negotiation.cc:407] Connection header received
0925 19:09:47.581576 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.581578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.581585 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.581639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.582843 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.582845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.582845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.582851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.582865 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.582870 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.582916 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.583007 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.583022 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:47.583299 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.580790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41617 (local address 127.27.174.65:35055)
0925 19:09:47.580977 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.581008 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.581474 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.581708 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.581712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.581721 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.582698 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.582698 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.582724 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.582753 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.582925 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.582929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.582932 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.583017 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.583076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260925 19:09:47.583817 670 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41617
I20260925 19:09:47.583915 670 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:47.584061 670 heartbeater.cc:511] Master 127.27.174.126:41617 requested a full tablet report, sending...
I20260925 19:09:47.584507 482 ts_manager.cc:194] Registered new tserver with Master: 011897d5829b4939bc6be6ab1b23805b (127.27.174.65:37173)
I20260925 19:09:47.588588 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 542
I20260925 19:09:47.588666 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:47.588796 28345 external_mini_cluster.cc:1428] Setting key 46da88a6f1bfaf136d5b7db7c5043ad2
I20260925 19:09:47.589612 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--builtin_ntp_servers=127.27.174.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:09:47.666908 673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:47.667140 673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:47.667176 673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:47.668390 673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:47.668457 673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:47.668496 673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:47.669817 673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:47.670078 673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:47.670363 673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:47.675420 678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.675453 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.675370 679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.675664 673 server_base.cc:1034] running on GCE node
I20260925 19:09:47.675871 673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:47.676060 673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:47.677191 673 hybrid_clock.cc:648] HybridClock initialized: now 1790363387677167 us; error 40 us; skew 500 ppm
I20260925 19:09:47.678395 673 webserver.cc:460] Webserver started at http://127.27.174.66:45117/ using document root <none> and password file <none>
I20260925 19:09:47.678580 673 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:47.678620 673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:47.678707 673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:47.679734 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "25a51fb7e1624081adfa95cba6a3deb8"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "447a237fc39181515dd7dc3fc6a4e230"
server_key_iv: "4d499769b97b5dd2681dffa46c47201d"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.680027 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "25a51fb7e1624081adfa95cba6a3deb8"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "447a237fc39181515dd7dc3fc6a4e230"
server_key_iv: "4d499769b97b5dd2681dffa46c47201d"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.681206 673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:47.681851 687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.682019 673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.682125 673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "25a51fb7e1624081adfa95cba6a3deb8"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "447a237fc39181515dd7dc3fc6a4e230"
server_key_iv: "4d499769b97b5dd2681dffa46c47201d"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.682194 673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:47.685491 673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:47.685724 673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:47.685855 673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:47.686043 673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:47.686326 673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:47.686357 673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.686407 673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:47.686435 673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.692063 673 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41849
I20260925 19:09:47.692121 800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41849 every 8 connection(s)
I20260925 19:09:47.692421 673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:47.693517 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 673
I20260925 19:09:47.693606 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:47.693836 28345 external_mini_cluster.cc:1428] Setting key 6e500955e9bbab7b77fdf615ec8ec81a
I20260925 19:09:47.694847 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--builtin_ntp_servers=127.27.174.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 19:09:47.695691 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.693100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:55479 (local address 127.27.174.126:41617)
0925 19:09:47.693160 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.693162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.693357 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:09:47.694015 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.694016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.694024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.694095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.695416 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.695417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.695417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.695422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.695438 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.695443 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.695504 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.695561 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.695575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:47.695858 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.693060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41617 (local address 127.27.174.66:55479)
0925 19:09:47.693226 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.693263 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.693888 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.694250 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.694255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.694264 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.695274 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.695274 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.695300 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.695327 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.695511 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.695513 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.695515 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.695566 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.695654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 19:09:47.696290 801 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41617
I20260925 19:09:47.696394 801 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:47.696537 801 heartbeater.cc:511] Master 127.27.174.126:41617 requested a full tablet report, sending...
I20260925 19:09:47.696869 482 ts_manager.cc:194] Registered new tserver with Master: 25a51fb7e1624081adfa95cba6a3deb8 (127.27.174.66:41849)
W20260925 19:09:47.774653 804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:47.774845 804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:47.774866 804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:47.776059 804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:47.776100 804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:47.776127 804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:47.777434 804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:47.777678 804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:47.777958 804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:47.782874 812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.782869 809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:47.782869 810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:47.783084 804 server_base.cc:1034] running on GCE node
I20260925 19:09:47.783365 804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:47.783565 804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:47.784734 804 hybrid_clock.cc:648] HybridClock initialized: now 1790363387784713 us; error 36 us; skew 500 ppm
I20260925 19:09:47.786017 804 webserver.cc:460] Webserver started at http://127.27.174.67:34231/ using document root <none> and password file <none>
I20260925 19:09:47.786275 804 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:47.786355 804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:47.786525 804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:47.787719 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "5f899c96ca4641a8bd2b795ec4030155"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "d235fbd9ce89cc83a937ac07df02de1c"
server_key_iv: "9bf4f6d1c37ea87aec8c4ae792d89186"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.788141 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "5f899c96ca4641a8bd2b795ec4030155"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "d235fbd9ce89cc83a937ac07df02de1c"
server_key_iv: "9bf4f6d1c37ea87aec8c4ae792d89186"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.789556 804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:47.790565 818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.790756 804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:47.790845 804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "5f899c96ca4641a8bd2b795ec4030155"
format_stamp: "Formatted at 2026-09-25 19:09:47 on dist-test-slave-fcdn"
server_key: "d235fbd9ce89cc83a937ac07df02de1c"
server_key_iv: "9bf4f6d1c37ea87aec8c4ae792d89186"
server_key_version: "encryptionkey@0"
I20260925 19:09:47.790925 804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:47.795987 804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:47.796310 804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:47.796440 804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:47.796659 804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:47.796979 804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:47.797015 804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.797035 804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:47.797048 804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:47.802304 804 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:36993
I20260925 19:09:47.802371 931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:36993 every 8 connection(s)
I20260925 19:09:47.802667 804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:47.805771 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.803404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50029 (local address 127.27.174.126:41617)
0925 19:09:47.803467 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.803470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.803761 (+ 291us) server_negotiation.cc:407] Connection header received
0925 19:09:47.804258 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.804260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.804266 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.804328 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.805509 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.805511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.805511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.805516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.805536 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.805541 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.805598 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.805684 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.805699 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:47.805948 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.803403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41617 (local address 127.27.174.67:50029)
0925 19:09:47.803589 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.803663 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.804135 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.804392 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.804396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.804405 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.805356 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.805357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.805382 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.805408 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.805612 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.805616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.805619 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.805683 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.805742 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260925 19:09:47.806448 932 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41617
I20260925 19:09:47.806519 932 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:47.806615 932 heartbeater.cc:511] Master 127.27.174.126:41617 requested a full tablet report, sending...
I20260925 19:09:47.806926 482 ts_manager.cc:194] Registered new tserver with Master: 5f899c96ca4641a8bd2b795ec4030155 (127.27.174.67:36993)
I20260925 19:09:47.809810 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 804
I20260925 19:09:47.809906 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:47.810078 28345 external_mini_cluster.cc:1428] Setting key f81fd1f3e4a3e6a9831d862df528f436
I20260925 19:09:47.810840 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:09:47.813099 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.812113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.27.174.126:41617)
0925 19:09:47.812166 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.812168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.812274 (+ 106us) server_negotiation.cc:407] Connection header received
0925 19:09:47.812332 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.812333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.812337 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.812375 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.812964 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.812965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.812965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.812967 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.812976 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.812979 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.813016 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.813049 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.813058 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:47.814468 482 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:47.816083 482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:47.819926 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36993 (local address 127.0.0.1:47376)
0925 19:09:47.818195 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.818231 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.818285 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.818647 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.818652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.818658 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.819396 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.819396 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.819412 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.819434 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.819773 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.819778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.819780 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.819841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.819854 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260925 19:09:47.820091 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.27.174.66:41849)
0925 19:09:47.818683 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.818687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.818739 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:09:47.818795 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.818798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.818807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.818886 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.819719 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.819725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.819725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.819729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.819760 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.819779 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.819843 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.819982 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.819999 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260925 19:09:47.820326 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37173 (local address 127.0.0.1:43168)
0925 19:09:47.818828 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.818836 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.818912 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.819518 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.819518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.819520 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.819921 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.819921 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.819927 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.819939 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.820200 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.820201 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.820202 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.820253 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.820266 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":91,"threads_started":1}
I20260925 19:09:47.820371 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.27.174.65:37173)
0925 19:09:47.818489 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.818493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.818889 (+ 396us) server_negotiation.cc:407] Connection header received
0925 19:09:47.818974 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.818982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.818993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.819067 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.820020 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.820028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.820028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.820034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.820072 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.820097 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.820155 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.820272 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.820287 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260925 19:09:47.819944 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41849 (local address 127.0.0.1:40730)
0925 19:09:47.818685 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.818695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.818744 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.819164 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.819165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.819167 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.819628 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.819628 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.819635 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.819648 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.819838 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.819838 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.819838 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.819884 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.819894 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":119,"threads_started":1}
I20260925 19:09:47.820719 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.818339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.27.174.67:36993)
0925 19:09:47.818494 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.818498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.818508 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:47.818530 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.818532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.818540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.818619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.819628 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.819635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.819636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.819640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.819674 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.819696 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.819782 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.820394 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.820541 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260925 19:09:47.821162 604 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27fe3bb03c4361957a5480038851a7 (DEFAULT_TABLE table=test-table [id=c5fa5734aa294c1fb6e91e1e77121555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.821241 735 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27fe3bb03c4361957a5480038851a7 (DEFAULT_TABLE table=test-table [id=c5fa5734aa294c1fb6e91e1e77121555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.821306 866 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27fe3bb03c4361957a5480038851a7 (DEFAULT_TABLE table=test-table [id=c5fa5734aa294c1fb6e91e1e77121555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:47.821522 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27fe3bb03c4361957a5480038851a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.821574 866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27fe3bb03c4361957a5480038851a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.821554 735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27fe3bb03c4361957a5480038851a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:47.823922 946 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Bootstrap starting.
I20260925 19:09:47.824638 946 tablet_bootstrap.cc:654] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.824811 947 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Bootstrap starting.
I20260925 19:09:47.824983 946 log.cc:824] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.825484 947 tablet_bootstrap.cc:654] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.825753 946 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: No bootstrap required, opened a new log
I20260925 19:09:47.825783 947 log.cc:824] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.825822 946 ts_tablet_manager.cc:1397] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:47.826036 948 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: Bootstrap starting.
I20260925 19:09:47.826432 947 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: No bootstrap required, opened a new log
I20260925 19:09:47.826502 947 ts_tablet_manager.cc:1397] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:47.826937 948 tablet_bootstrap.cc:654] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:47.827250 948 log.cc:824] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:47.828087 948 tablet_bootstrap.cc:492] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: No bootstrap required, opened a new log
I20260925 19:09:47.828169 948 ts_tablet_manager.cc:1397] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:47.828537 946 raft_consensus.cc:348] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.828701 946 raft_consensus.cc:374] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.828763 946 raft_consensus.cc:729] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a51fb7e1624081adfa95cba6a3deb8, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.828663 947 raft_consensus.cc:348] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.828840 947 raft_consensus.cc:374] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.828883 947 raft_consensus.cc:729] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f899c96ca4641a8bd2b795ec4030155, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.828902 946 consensus_queue.cc:260] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.829025 947 consensus_queue.cc:260] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.829180 946 ts_tablet_manager.cc:1428] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 19:09:47.829304 801 heartbeater.cc:503] Master 127.27.174.126:41617 was elected leader, sending a full tablet report...
I20260925 19:09:47.829308 947 ts_tablet_manager.cc:1428] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:47.829444 932 heartbeater.cc:503] Master 127.27.174.126:41617 was elected leader, sending a full tablet report...
I20260925 19:09:47.831049 948 raft_consensus.cc:348] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.831167 948 raft_consensus.cc:374] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:47.831187 948 raft_consensus.cc:729] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011897d5829b4939bc6be6ab1b23805b, State: Initialized, Role: FOLLOWER
I20260925 19:09:47.831321 948 consensus_queue.cc:260] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.831573 948 ts_tablet_manager.cc:1428] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:47.831580 670 heartbeater.cc:503] Master 127.27.174.126:41617 was elected leader, sending a full tablet report...
I20260925 19:09:47.858620 954 raft_consensus.cc:482] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:47.858738 954 raft_consensus.cc:504] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.859117 954 leader_election.cc:288] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f899c96ca4641a8bd2b795ec4030155 (127.27.174.67:36993), 25a51fb7e1624081adfa95cba6a3deb8 (127.27.174.66:41849)
I20260925 19:09:47.860358 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.859287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41849 (local address 127.27.174.65:56369)
0925 19:09:47.859347 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.859359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.859394 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.859533 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.859534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.859537 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.860001 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.860002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.860008 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.860036 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.860210 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.860212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.860213 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.860264 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.860278 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 19:09:47.860381 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.859333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:56369 (local address 127.27.174.66:41849)
0925 19:09:47.859397 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.859399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.859431 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:09:47.859457 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.859458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.859464 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.859508 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.860107 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.860108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.860108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.860113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.860128 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.860134 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.860201 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.860287 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.860299 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:09:47.860620 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.859455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47991 (local address 127.27.174.67:36993)
0925 19:09:47.859496 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.859497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.859585 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:09:47.859676 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.859677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.859682 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.859740 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.860354 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.860355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.860356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.860360 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.860376 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.860380 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.860441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.860546 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.860559 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:47.860674 755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27fe3bb03c4361957a5480038851a7" candidate_uuid: "011897d5829b4939bc6be6ab1b23805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a51fb7e1624081adfa95cba6a3deb8" is_pre_election: true
I20260925 19:09:47.860814 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.859367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36993 (local address 127.27.174.65:47991)
0925 19:09:47.859511 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.859521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:47.859587 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:47.859768 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:47.859769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:47.859771 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:47.860256 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:09:47.860257 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:47.860263 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:09:47.860274 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:47.860471 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:47.860472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:47.860473 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:47.860716 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 19:09:47.860730 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260925 19:09:47.860818 755 raft_consensus.cc:2454] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011897d5829b4939bc6be6ab1b23805b in term 0.
I20260925 19:09:47.861061 557 leader_election.cc:302] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011897d5829b4939bc6be6ab1b23805b, 25a51fb7e1624081adfa95cba6a3deb8; no voters:
I20260925 19:09:47.861112 886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27fe3bb03c4361957a5480038851a7" candidate_uuid: "011897d5829b4939bc6be6ab1b23805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f899c96ca4641a8bd2b795ec4030155" is_pre_election: true
I20260925 19:09:47.861204 954 raft_consensus.cc:2781] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:47.861256 954 raft_consensus.cc:482] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:47.861258 886 raft_consensus.cc:2454] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011897d5829b4939bc6be6ab1b23805b in term 0.
I20260925 19:09:47.861289 954 raft_consensus.cc:3037] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.861997 954 raft_consensus.cc:504] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.862139 954 leader_election.cc:288] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [CANDIDATE]: Term 1 election: Requested vote from peers 5f899c96ca4641a8bd2b795ec4030155 (127.27.174.67:36993), 25a51fb7e1624081adfa95cba6a3deb8 (127.27.174.66:41849)
I20260925 19:09:47.862282 886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27fe3bb03c4361957a5480038851a7" candidate_uuid: "011897d5829b4939bc6be6ab1b23805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f899c96ca4641a8bd2b795ec4030155"
I20260925 19:09:47.862355 886 raft_consensus.cc:3037] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.863024 886 raft_consensus.cc:2454] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011897d5829b4939bc6be6ab1b23805b in term 1.
I20260925 19:09:47.863135 755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27fe3bb03c4361957a5480038851a7" candidate_uuid: "011897d5829b4939bc6be6ab1b23805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a51fb7e1624081adfa95cba6a3deb8"
I20260925 19:09:47.863184 559 leader_election.cc:302] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011897d5829b4939bc6be6ab1b23805b, 5f899c96ca4641a8bd2b795ec4030155; no voters:
I20260925 19:09:47.863217 755 raft_consensus.cc:3037] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:47.863310 954 raft_consensus.cc:2781] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:47.863467 954 raft_consensus.cc:686] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [term 1 LEADER]: Becoming Leader. State: Replica: 011897d5829b4939bc6be6ab1b23805b, State: Running, Role: LEADER
I20260925 19:09:47.863555 954 consensus_queue.cc:237] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } }
I20260925 19:09:47.864121 755 raft_consensus.cc:2454] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011897d5829b4939bc6be6ab1b23805b in term 1.
I20260925 19:09:47.864355 482 catalog_manager.cc:5495] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b reported cstate change: term changed from 0 to 1, leader changed from <none> to 011897d5829b4939bc6be6ab1b23805b (127.27.174.65). New cstate: current_term: 1 leader_uuid: "011897d5829b4939bc6be6ab1b23805b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011897d5829b4939bc6be6ab1b23805b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37173 } health_report { overall_health: HEALTHY } } }
I20260925 19:09:47.868922 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.867906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.27.174.65:37173)
0925 19:09:47.867948 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:47.867950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:47.867985 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:09:47.868060 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:47.868061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:47.868078 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:47.868141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:47.868718 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:47.868719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:47.868719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:47.868723 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:47.868736 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:47.868740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:47.868794 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:47.868847 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:09:47.868859 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:47.870390 886 raft_consensus.cc:1261] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 1 FOLLOWER]: Refusing update from remote peer 011897d5829b4939bc6be6ab1b23805b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:47.870414 755 raft_consensus.cc:1261] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 1 FOLLOWER]: Refusing update from remote peer 011897d5829b4939bc6be6ab1b23805b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:47.870702 956 consensus_queue.cc:1035] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f899c96ca4641a8bd2b795ec4030155" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:47.870836 954 consensus_queue.cc:1035] T 8c27fe3bb03c4361957a5480038851a7 P 011897d5829b4939bc6be6ab1b23805b [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a51fb7e1624081adfa95cba6a3deb8" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:47.873809 960 mvcc.cc:205] Tried to move back new op lower bound from 7333328436715016192 to 7333328436689985536. Current Snapshot: MvccSnapshot[applied={T|T < 7333328436715016192}]
I20260925 19:09:47.875061 482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:47.875142 482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:47.875811 482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 823ddb6725084ccda53b92d8b4956e3f: Sending DeleteTablet for 3 replicas of tablet 8c27fe3bb03c4361957a5480038851a7
I20260925 19:09:47.876112 866 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c27fe3bb03c4361957a5480038851a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='slave'} at 127.0.0.1:47376
I20260925 19:09:47.876134 735 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c27fe3bb03c4361957a5480038851a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='slave'} at 127.0.0.1:40730
I20260925 19:09:47.876266 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c27fe3bb03c4361957a5480038851a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:47 UTC) from {username='slave'} at 127.0.0.1:43168
I20260925 19:09:47.876360 968 tablet_replica.cc:330] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: stopping tablet replica
I20260925 19:09:47.876449 968 raft_consensus.cc:2229] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:47.876550 968 raft_consensus.cc:2258] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:47.876564 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 542
I20260925 19:09:47.877467 968 ts_tablet_manager.cc:1905] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:47.878885 969 tablet_replica.cc:330] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: stopping tablet replica
I20260925 19:09:47.878966 969 raft_consensus.cc:2229] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:47.879086 969 raft_consensus.cc:2258] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:47.879249 968 ts_tablet_manager.cc:1918] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:47.879320 968 log.cc:1196] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/8c27fe3bb03c4361957a5480038851a7
I20260925 19:09:47.879518 969 ts_tablet_manager.cc:1905] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:47.879575 968 ts_tablet_manager.cc:1939] T 8c27fe3bb03c4361957a5480038851a7 P 5f899c96ca4641a8bd2b795ec4030155: Deleting consensus metadata
I20260925 19:09:47.879982 468 catalog_manager.cc:4841] TS 5f899c96ca4641a8bd2b795ec4030155 (127.27.174.67:36993): tablet 8c27fe3bb03c4361957a5480038851a7 (table test-table [id=c5fa5734aa294c1fb6e91e1e77121555]) successfully deleted
I20260925 19:09:47.880960 969 ts_tablet_manager.cc:1918] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:47.881044 969 log.cc:1196] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/8c27fe3bb03c4361957a5480038851a7
I20260925 19:09:47.881289 969 ts_tablet_manager.cc:1939] T 8c27fe3bb03c4361957a5480038851a7 P 25a51fb7e1624081adfa95cba6a3deb8: Deleting consensus metadata
I20260925 19:09:47.881672 466 catalog_manager.cc:4841] TS 25a51fb7e1624081adfa95cba6a3deb8 (127.27.174.66:41849): tablet 8c27fe3bb03c4361957a5480038851a7 (table test-table [id=c5fa5734aa294c1fb6e91e1e77121555]) successfully deleted
W20260925 19:09:47.883903 469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:37173 (error 108)
I20260925 19:09:47.884101 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 673
I20260925 19:09:47.885329 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:47.885183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37173 (local address 127.0.0.1:43184)
0925 19:09:47.885252 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:47.885294 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 19:09:47.885488 469 catalog_manager.cc:4584] TS 011897d5829b4939bc6be6ab1b23805b (127.27.174.65:37173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c27fe3bb03c4361957a5480038851a7: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37173: connect: Connection refused (error 111)
I20260925 19:09:47.889135 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 804
I20260925 19:09:47.893594 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 451
2026-09-25T19:09:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (578 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 19:09:47.913177 28345 mini_oidc.cc:152] Starting JWKS server
I20260925 19:09:47.914171 28345 webserver.cc:460] Webserver started at https://127.0.0.1:39775/ using document root <none> and password file <none>
I20260925 19:09:47.914291 28345 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:09:47.914479 28345 webserver.cc:460] Webserver started at http://127.0.0.1:39967/ using document root <none> and password file <none>
2026-09-25T19:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:47Z Disabled control of system clock
I20260925 19:09:47.922003 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46119
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39775/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/testchainca.cert with env {}
W20260925 19:09:48.000442 977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:48.000648 977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:48.000669 977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:48.001773 977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:48.001806 977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:48.001818 977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:48.001830 977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:48.001843 977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:09:48.001855 977 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:09:48.001866 977 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39775/jwks.json
I20260925 19:09:48.003307 977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39775/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:48.003500 977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:48.003685 977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:48.008435 985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:48.008466 977 server_base.cc:1034] running on GCE node
W20260925 19:09:48.008435 983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.008428 982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:48.008867 977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:48.009181 977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:48.010387 977 hybrid_clock.cc:648] HybridClock initialized: now 1790363388010370 us; error 26 us; skew 500 ppm
I20260925 19:09:48.021859 977 webserver.cc:460] Webserver started at http://127.27.174.126:35601/ using document root <none> and password file <none>
I20260925 19:09:48.022122 977 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:48.022208 977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:48.022359 977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:48.023445 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "222423e12b944308ad207a3e4ba7d666"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "eaa11e50eb86f947f78db1a21d1de095"
server_key_iv: "1bd174c4821ff646ae259265eb56b46c"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.023845 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "222423e12b944308ad207a3e4ba7d666"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "eaa11e50eb86f947f78db1a21d1de095"
server_key_iv: "1bd174c4821ff646ae259265eb56b46c"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.025141 977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:48.025890 991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:48.026110 977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:48.026221 977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "222423e12b944308ad207a3e4ba7d666"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "eaa11e50eb86f947f78db1a21d1de095"
server_key_iv: "1bd174c4821ff646ae259265eb56b46c"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.026302 977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:48.037207 977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:48.037473 977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:48.037616 977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:48.041641 977 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46119
I20260925 19:09:48.041669 1043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46119 every 8 connection(s)
I20260925 19:09:48.041983 977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:48.042874 1044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:48.045177 1044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666: Bootstrap starting.
I20260925 19:09:48.045733 1044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:48.045964 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 977
I20260925 19:09:48.046067 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:48.046087 1044 log.cc:824] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:48.046263 28345 external_mini_cluster.cc:1428] Setting key c08b347ac1acd36ddda79b883737cabf
I20260925 19:09:48.046984 1044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666: No bootstrap required, opened a new log
I20260925 19:09:48.049667 1044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } }
I20260925 19:09:48.049820 1044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:48.049865 1044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222423e12b944308ad207a3e4ba7d666, State: Initialized, Role: FOLLOWER
I20260925 19:09:48.049986 1044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } }
I20260925 19:09:48.050083 1044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:48.050124 1044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:48.050171 1044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:48.050825 1044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } }
I20260925 19:09:48.050925 1044 leader_election.cc:302] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 222423e12b944308ad207a3e4ba7d666; no voters:
I20260925 19:09:48.051079 1044 leader_election.cc:288] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:48.051148 1049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:48.051307 1049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [term 1 LEADER]: Becoming Leader. State: Replica: 222423e12b944308ad207a3e4ba7d666, State: Running, Role: LEADER
I20260925 19:09:48.051334 1044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:48.051403 1049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } }
I20260925 19:09:48.051802 1051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222423e12b944308ad207a3e4ba7d666. Latest consensus state: current_term: 1 leader_uuid: "222423e12b944308ad207a3e4ba7d666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } } }
I20260925 19:09:48.052496 1051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.051980 1050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "222423e12b944308ad207a3e4ba7d666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222423e12b944308ad207a3e4ba7d666" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46119 } } }
I20260925 19:09:48.052572 1050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.052716 1063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:48.053081 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.046716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.27.174.126:46119)
0925 19:09:48.046933 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.046942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.046974 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:09:48.047481 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.047490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.047573 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.047747 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:48.047957 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.049386 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.050410 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.051037 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.051189 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.052016 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:48.052020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:48.052021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:48.052217 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:48.052264 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:48.052281 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:48.052713 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:48.052801 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 19:09:48.052878 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260925 19:09:48.053210 1063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:48.054032 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:48.054741 1063 catalog_manager.cc:1347] Generated new cluster ID: 55288da4049e4ff89fb29fb643b4ab54
I20260925 19:09:48.054790 1063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:48.063405 1063 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:48.064092 1063 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:48.076159 1063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 222423e12b944308ad207a3e4ba7d666: Generated new TSK 0
I20260925 19:09:48.076382 1063 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:09:48.101825 1074 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 19:09:48.101923 1074 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 19:09:48.102173 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.087208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.27.174.126:46119)
0925 19:09:48.087252 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.087255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.087361 (+ 106us) server_negotiation.cc:407] Connection header received
0925 19:09:48.087450 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.087450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.087460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.087516 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:09:48.087728 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.088234 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.089345 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.089525 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.089576 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.089838 (+ 262us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:09:48.101961 (+ 12123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:09:48.102110 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 19:09:48.102246 1048 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 19:09:48.102540 1073 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:48.087093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46119 (local address 127.0.0.1:54256)
0925 19:09:48.087280 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:48.087292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:48.087364 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:48.087540 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:48.087541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:48.087543 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:09:48.087646 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.087648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.088309 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.088310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.089262 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.089265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.089609 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.089610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.089686 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.089708 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:09:48.102102 (+ 12394us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:09:48.102211 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:46119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.174.126:46119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 19:09:48.102965 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 977
2026-09-25T19:09:48Z chronyd exiting
I20260925 19:09:48.139035 28345 test_util.cc:175] -----------------------------------------------
I20260925 19:09:48.139124 28345 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363366557183-28345-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 19:09:48.140054 28345 mini_oidc.cc:152] Starting JWKS server
I20260925 19:09:48.140830 28345 webserver.cc:460] Webserver started at https://127.0.0.1:41909/ using document root <none> and password file <none>
I20260925 19:09:48.140877 28345 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:09:48.141076 28345 webserver.cc:460] Webserver started at http://127.0.0.1:40069/ using document root <none> and password file <none>
2026-09-25T19:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:48Z Disabled control of system clock
I20260925 19:09:48.149355 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38857
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41909/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/testchainca.cert with env {}
W20260925 19:09:48.229947 1080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:48.230139 1080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:48.230160 1080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:48.231228 1080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:48.231261 1080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:48.231273 1080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:48.231285 1080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:48.231297 1080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:09:48.231309 1080 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:09:48.231320 1080 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41909/jwks.json
I20260925 19:09:48.232758 1080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41909/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:48.232990 1080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:48.233170 1080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:48.237949 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.237985 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.237990 1088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:48.238116 1080 server_base.cc:1034] running on GCE node
I20260925 19:09:48.238373 1080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:48.238595 1080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:48.239773 1080 hybrid_clock.cc:648] HybridClock initialized: now 1790363388239759 us; error 25 us; skew 500 ppm
I20260925 19:09:48.241036 1080 webserver.cc:460] Webserver started at http://127.27.174.126:37161/ using document root <none> and password file <none>
I20260925 19:09:48.241293 1080 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:48.241379 1080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:48.241515 1080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:48.242525 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "204c23d37d0e4710a2528655d60a6ff2"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "b45eba75491096afd6c85b59aface7d3"
server_key_iv: "ee6e3b93c78fbe7fac887a197b6e92b2"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.242915 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "204c23d37d0e4710a2528655d60a6ff2"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "b45eba75491096afd6c85b59aface7d3"
server_key_iv: "ee6e3b93c78fbe7fac887a197b6e92b2"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.244310 1080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:48.244987 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:48.245153 1080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:48.245249 1080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "204c23d37d0e4710a2528655d60a6ff2"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "b45eba75491096afd6c85b59aface7d3"
server_key_iv: "ee6e3b93c78fbe7fac887a197b6e92b2"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.245327 1080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:48.264405 1080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:48.264729 1080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:48.264889 1080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:48.268833 1080 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38857
I20260925 19:09:48.268862 1146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38857 every 8 connection(s)
I20260925 19:09:48.269207 1080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:48.269866 1147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:48.272496 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2: Bootstrap starting.
I20260925 19:09:48.273012 1147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:48.273244 1147 log.cc:824] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:48.273538 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1080
I20260925 19:09:48.273649 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:48.273818 28345 external_mini_cluster.cc:1428] Setting key 9e74905f633abc85fce271738586cdf9
I20260925 19:09:48.273993 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2: No bootstrap required, opened a new log
I20260925 19:09:48.275843 1147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } }
I20260925 19:09:48.275975 1147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:48.275996 1147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204c23d37d0e4710a2528655d60a6ff2, State: Initialized, Role: FOLLOWER
I20260925 19:09:48.276150 1147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } }
I20260925 19:09:48.276217 1147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:48.276240 1147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:48.276310 1147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:48.277037 1147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } }
I20260925 19:09:48.277141 1147 leader_election.cc:302] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 204c23d37d0e4710a2528655d60a6ff2; no voters:
I20260925 19:09:48.277287 1147 leader_election.cc:288] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:48.277340 1152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:48.277489 1152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 204c23d37d0e4710a2528655d60a6ff2, State: Running, Role: LEADER
I20260925 19:09:48.277508 1147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:48.277573 1152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } }
I20260925 19:09:48.277877 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "204c23d37d0e4710a2528655d60a6ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } } }
I20260925 19:09:48.277978 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.278177 1160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:48.278203 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204c23d37d0e4710a2528655d60a6ff2. Latest consensus state: current_term: 1 leader_uuid: "204c23d37d0e4710a2528655d60a6ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204c23d37d0e4710a2528655d60a6ff2" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38857 } } }
I20260925 19:09:48.278340 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.278690 1160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:48.280189 1160 catalog_manager.cc:1347] Generated new cluster ID: 621b44839f7d48e1a86b01eac70c21b6
I20260925 19:09:48.280241 1160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:48.281474 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.274295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.27.174.126:38857)
0925 19:09:48.274511 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.274522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.274553 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:09:48.275825 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.275833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.275895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.276053 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:48.276359 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.278162 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.279352 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.279940 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.280373 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.280757 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:48.280761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:48.280761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:48.280962 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:48.281002 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:48.281015 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:48.281082 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:48.281188 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 19:09:48.281262 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260925 19:09:48.282307 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:48.287717 1160 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:48.288184 1160 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:48.299162 1160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 204c23d37d0e4710a2528655d60a6ff2: Generated new TSK 0
I20260925 19:09:48.299365 1160 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:09:48.319823 1177 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 19:09:48.319900 1177 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 19:09:48.320268 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.305744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.27.174.126:38857)
0925 19:09:48.305810 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.305812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.305951 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:09:48.306026 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.306027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.306039 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.306133 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:09:48.306270 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.306757 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.307571 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.307802 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.307884 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.308073 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:09:48.320043 (+ 11970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:09:48.320197 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 19:09:48.320326 1151 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 19:09:48.320340 1176 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:48.305695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38857 (local address 127.0.0.1:37574)
0925 19:09:48.305866 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:48.305877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:48.305960 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:48.306133 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:48.306134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:48.306136 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:09:48.306194 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.306196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.306862 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.306863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.307477 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.307479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.307885 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.307885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.307953 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.307958 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:09:48.320148 (+ 12190us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:09:48.320240 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:38857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260925 19:09:48.320688 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1080
2026-09-25T19:09:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 19:09:48.357914 28345 mini_oidc.cc:152] Starting JWKS server
I20260925 19:09:48.358590 28345 webserver.cc:460] Webserver started at https://127.0.0.1:33041/ using document root <none> and password file <none>
I20260925 19:09:48.358633 28345 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:09:48.358844 28345 webserver.cc:460] Webserver started at http://127.0.0.1:36639/ using document root <none> and password file <none>
2026-09-25T19:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:48Z Disabled control of system clock
I20260925 19:09:48.366250 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44555
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44555
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:09:48.444595 1183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:48.444820 1183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:48.444864 1183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:48.445994 1183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:48.446064 1183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:48.446085 1183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:48.446105 1183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:48.446136 1183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:09:48.446166 1183 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:09:48.446188 1183 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33041/jwks.json
I20260925 19:09:48.447508 1183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:48.447795 1183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:48.448076 1183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:48.452677 1188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.452713 1189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.452708 1191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:48.452916 1183 server_base.cc:1034] running on GCE node
I20260925 19:09:48.453152 1183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:48.453414 1183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:48.454588 1183 hybrid_clock.cc:648] HybridClock initialized: now 1790363388454562 us; error 47 us; skew 500 ppm
I20260925 19:09:48.455853 1183 webserver.cc:460] Webserver started at http://127.27.174.126:34841/ using document root <none> and password file <none>
I20260925 19:09:48.456115 1183 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:48.456193 1183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:48.456341 1183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:48.457348 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "670772ca8e5d45b79f47ac369d41f675"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "9aa6039a0901ecf436696d49160f9c1e"
server_key_iv: "8004a07dbf44322467160f19c9ec53b7"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.457697 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "670772ca8e5d45b79f47ac369d41f675"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "9aa6039a0901ecf436696d49160f9c1e"
server_key_iv: "8004a07dbf44322467160f19c9ec53b7"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.459028 1183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:48.459784 1197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:48.459952 1183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:48.460048 1183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "670772ca8e5d45b79f47ac369d41f675"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "9aa6039a0901ecf436696d49160f9c1e"
server_key_iv: "8004a07dbf44322467160f19c9ec53b7"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.460132 1183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:48.470867 1183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:48.471132 1183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:48.471282 1183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:48.475086 1183 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44555
I20260925 19:09:48.475137 1249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44555 every 8 connection(s)
I20260925 19:09:48.475481 1183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:48.476205 1250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:48.478623 1250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675: Bootstrap starting.
I20260925 19:09:48.479126 1250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:48.479350 1250 log.cc:824] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:48.480070 1250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675: No bootstrap required, opened a new log
I20260925 19:09:48.480144 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1183
I20260925 19:09:48.480232 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:48.480391 28345 external_mini_cluster.cc:1428] Setting key b08c29b0232bc6de1c4347633c25b634
I20260925 19:09:48.483136 1250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } }
I20260925 19:09:48.483299 1250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:48.483327 1250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670772ca8e5d45b79f47ac369d41f675, State: Initialized, Role: FOLLOWER
I20260925 19:09:48.483434 1250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } }
I20260925 19:09:48.483498 1250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:48.483526 1250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:48.483575 1250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:48.484166 1250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } }
I20260925 19:09:48.484256 1250 leader_election.cc:302] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 670772ca8e5d45b79f47ac369d41f675; no voters:
I20260925 19:09:48.484448 1250 leader_election.cc:288] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:48.484501 1255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:48.484696 1250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:48.484761 1255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [term 1 LEADER]: Becoming Leader. State: Replica: 670772ca8e5d45b79f47ac369d41f675, State: Running, Role: LEADER
I20260925 19:09:48.484846 1255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } }
I20260925 19:09:48.485193 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "670772ca8e5d45b79f47ac369d41f675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } } }
I20260925 19:09:48.485201 1255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 670772ca8e5d45b79f47ac369d41f675. Latest consensus state: current_term: 1 leader_uuid: "670772ca8e5d45b79f47ac369d41f675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670772ca8e5d45b79f47ac369d41f675" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44555 } } }
I20260925 19:09:48.485350 1255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.485505 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.485597 1264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:48.486063 1264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:48.487144 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.480883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.27.174.126:44555)
0925 19:09:48.481095 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.481104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.481134 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:09:48.481727 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.481736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.481812 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.481962 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:48.482148 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.483549 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.484466 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.485078 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.486066 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.486304 (+ 238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:48.486307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:48.486308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:48.486516 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:48.486547 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:48.486557 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:48.486636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:48.486836 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 19:09:48.486917 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260925 19:09:48.488035 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:48.488732 1264 catalog_manager.cc:1347] Generated new cluster ID: d52b2cdf84a747eeb2ea4bf1757a0ce8
I20260925 19:09:48.488785 1264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:48.501449 1264 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:48.502128 1264 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:48.516356 1264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 670772ca8e5d45b79f47ac369d41f675: Generated new TSK 0
I20260925 19:09:48.516567 1264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:48.528800 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.525128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.27.174.126:44555)
0925 19:09:48.525194 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.525197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.525244 (+ 47us) server_negotiation.cc:407] Connection header received
0925 19:09:48.525335 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.525336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.525346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.525404 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:09:48.525570 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.526063 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.527188 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.527381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.527429 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.527677 (+ 248us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:09:48.528573 (+ 896us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:09:48.528710 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 19:09:48.528855 1254 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 19:09:48.528893 1279 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:48.525021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44555 (local address 127.0.0.1:40026)
0925 19:09:48.525170 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:48.525193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:48.525263 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:48.525428 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:48.525429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:48.525431 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:09:48.525502 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.525505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.526150 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.526151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.527105 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.527107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.527468 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.527469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.527535 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.527542 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:09:48.528680 (+ 1138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:09:48.528787 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:44555: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":87,"threads_started":1}
I20260925 19:09:48.529119 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1183
E20260925 19:09:48.539762 1280 webserver.cc:533] Webserver: sslize failed:
E20260925 19:09:48.539822 1280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T19:09:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T19:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:48Z Disabled control of system clock
I20260925 19:09:48.573307 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38163
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38163
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/testchainca.cert with env {}
W20260925 19:09:48.652994 1284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:48.653182 1284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:48.653203 1284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:48.654326 1284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:48.654376 1284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:48.654399 1284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:48.654417 1284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:48.654433 1284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:09:48.654451 1284 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:09:48.654479 1284 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 19:09:48.656379 1284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:48.656569 1284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:48.656782 1284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:48.661425 1289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.661437 1292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:48.661440 1290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:48.661547 1284 server_base.cc:1034] running on GCE node
I20260925 19:09:48.661855 1284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:48.662140 1284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:48.663300 1284 hybrid_clock.cc:648] HybridClock initialized: now 1790363388663288 us; error 47 us; skew 500 ppm
I20260925 19:09:48.664533 1284 webserver.cc:460] Webserver started at http://127.27.174.126:38289/ using document root <none> and password file <none>
I20260925 19:09:48.664793 1284 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:48.664865 1284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:48.665009 1284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:48.666028 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "19ff0b064d7746af81396d1be6b2f123"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "d8996cf0a2d803b527f49db8501281c8"
server_key_iv: "786a124f28523be73aeb568c6c52cb04"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.666407 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "19ff0b064d7746af81396d1be6b2f123"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "d8996cf0a2d803b527f49db8501281c8"
server_key_iv: "786a124f28523be73aeb568c6c52cb04"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.667752 1284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:48.668432 1298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:48.668617 1284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:48.668713 1284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "19ff0b064d7746af81396d1be6b2f123"
format_stamp: "Formatted at 2026-09-25 19:09:48 on dist-test-slave-fcdn"
server_key: "d8996cf0a2d803b527f49db8501281c8"
server_key_iv: "786a124f28523be73aeb568c6c52cb04"
server_key_version: "encryptionkey@0"
I20260925 19:09:48.668792 1284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:48.686426 1284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:48.686728 1284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:48.686877 1284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:48.690821 1284 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38163
I20260925 19:09:48.690856 1350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38163 every 8 connection(s)
I20260925 19:09:48.691169 1284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:48.692162 1351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:48.694357 1351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123: Bootstrap starting.
I20260925 19:09:48.694955 1351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:48.695240 1351 log.cc:824] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:48.695953 1351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123: No bootstrap required, opened a new log
I20260925 19:09:48.697286 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1284
I20260925 19:09:48.697360 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:48.697546 28345 external_mini_cluster.cc:1428] Setting key f2b346da88f2299f0ddeb7927a38abe2
I20260925 19:09:48.697643 1351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } }
I20260925 19:09:48.697850 1351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:48.697893 1351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ff0b064d7746af81396d1be6b2f123, State: Initialized, Role: FOLLOWER
I20260925 19:09:48.698046 1351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } }
I20260925 19:09:48.698148 1351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:48.698190 1351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:48.698233 1351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:48.698741 1351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } }
I20260925 19:09:48.698858 1351 leader_election.cc:302] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19ff0b064d7746af81396d1be6b2f123; no voters:
I20260925 19:09:48.699034 1351 leader_election.cc:288] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:48.699198 1356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:48.699303 1351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:48.699337 1356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [term 1 LEADER]: Becoming Leader. State: Replica: 19ff0b064d7746af81396d1be6b2f123, State: Running, Role: LEADER
I20260925 19:09:48.699414 1356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } }
I20260925 19:09:48.699810 1356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ff0b064d7746af81396d1be6b2f123. Latest consensus state: current_term: 1 leader_uuid: "19ff0b064d7746af81396d1be6b2f123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } } }
I20260925 19:09:48.699894 1356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.699976 1357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19ff0b064d7746af81396d1be6b2f123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff0b064d7746af81396d1be6b2f123" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38163 } } }
I20260925 19:09:48.700085 1357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:48.700086 1364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:48.700731 1364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:48.702577 1364 catalog_manager.cc:1347] Generated new cluster ID: 917288defc5c447986b127ecd98f43ff
I20260925 19:09:48.702670 1364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:48.703747 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.698032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.27.174.126:38163)
0925 19:09:48.698274 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.698283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.698311 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:09:48.698421 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.698426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.698481 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.698621 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:48.698854 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.700215 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.701363 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.701968 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.702082 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.702928 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:48.702932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:48.702932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:48.703131 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:48.703172 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:48.703186 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:48.703244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:48.703434 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:09:48.703509 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260925 19:09:48.704609 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:48.721380 1364 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:48.721940 1364 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:48.728060 1364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19ff0b064d7746af81396d1be6b2f123: Generated new TSK 0
I20260925 19:09:48.728242 1364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:48.932301 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:48.734938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.27.174.126:38163)
0925 19:09:48.734992 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:48.734995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:48.735087 (+ 92us) server_negotiation.cc:407] Connection header received
0925 19:09:48.735154 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:48.735154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:48.735168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:48.735239 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:09:48.735420 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.736241 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.737018 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.737335 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.737403 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.737544 (+ 141us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:09:48.932059 (+194515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:09:48.932243 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60966: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 19:09:48.932396 1355 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60966: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 19:09:48.932399 1380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:48.734854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38163 (local address 127.0.0.1:60966)
0925 19:09:48.735025 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:48.735033 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:48.735082 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:48.735257 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:48.735258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:48.735261 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:09:48.735331 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.735333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.736326 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.736327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.736941 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:48.736943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:48.737414 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:48.737415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:48.737453 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:48.737458 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:09:48.932212 (+194754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:09:48.932325 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:38163: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260925 19:09:48.932799 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1284
2026-09-25T19:09:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (384 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:49 dist-test-slave-fcdn krb5kdc[1392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:49 dist-test-slave-fcdn krb5kdc[1392](info): set up 2 sockets
Sep 25 19:09:49 dist-test-slave-fcdn krb5kdc[1392](info): commencing operation
krb5kdc: starting...
W20260925 19:09:50.147933 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:09:50.267087 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42383
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:50.348402 1405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:50.348649 1405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:50.348675 1405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:50.349864 1405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:50.349905 1405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:50.349920 1405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:50.349932 1405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:50.349944 1405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:50.351881 1405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:50.352150 1405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:50.352399 1405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:50.357252 1413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.357270 1411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.357272 1410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:50.357483 1405 server_base.cc:1034] running on GCE node
I20260925 19:09:50.357765 1405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:50.358072 1405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:50.359248 1405 hybrid_clock.cc:648] HybridClock initialized: now 1790363390359236 us; error 31 us; skew 500 ppm
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.361948 1405 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:09:50.362363 1405 webserver.cc:460] Webserver started at http://127.27.174.126:37299/ using document root <none> and password file <none>
I20260925 19:09:50.362624 1405 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:50.362691 1405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:50.362830 1405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:50.363940 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "425408bdcf26471ebc43545dabdb9986"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "33c629c80bf7b7dab81c2425305d64d1"
server_key_iv: "9379fa47ba6cc91b54f6ac98b031b9bd"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.364281 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "425408bdcf26471ebc43545dabdb9986"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "33c629c80bf7b7dab81c2425305d64d1"
server_key_iv: "9379fa47ba6cc91b54f6ac98b031b9bd"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.365533 1405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:50.366303 1420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.366485 1405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:50.366591 1405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "425408bdcf26471ebc43545dabdb9986"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "33c629c80bf7b7dab81c2425305d64d1"
server_key_iv: "9379fa47ba6cc91b54f6ac98b031b9bd"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.366664 1405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:50.378609 1405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:50.380839 1405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:50.380995 1405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:50.384802 1405 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:42383
I20260925 19:09:50.384852 1472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:42383 every 8 connection(s)
I20260925 19:09:50.385200 1405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:50.386076 1473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:50.388378 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986: Bootstrap starting.
I20260925 19:09:50.388938 1473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:50.389286 1473 log.cc:824] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:50.389933 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986: No bootstrap required, opened a new log
I20260925 19:09:50.391499 1473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } }
I20260925 19:09:50.391587 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1405
I20260925 19:09:50.391688 1473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:50.391736 1473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425408bdcf26471ebc43545dabdb9986, State: Initialized, Role: FOLLOWER
I20260925 19:09:50.391762 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:50.391887 1473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } }
I20260925 19:09:50.391942 28345 external_mini_cluster.cc:1428] Setting key 19ec03e221dd9df092360e0f1a774efb
I20260925 19:09:50.391971 1473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:50.392011 1473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:50.392062 1473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:50.392990 1473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } }
I20260925 19:09:50.393123 1473 leader_election.cc:302] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 425408bdcf26471ebc43545dabdb9986; no voters:
I20260925 19:09:50.393319 1473 leader_election.cc:288] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:50.393353 1478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:50.393451 1478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [term 1 LEADER]: Becoming Leader. State: Replica: 425408bdcf26471ebc43545dabdb9986, State: Running, Role: LEADER
I20260925 19:09:50.393530 1478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } }
I20260925 19:09:50.393577 1473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:50.394091 1479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "425408bdcf26471ebc43545dabdb9986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } } }
I20260925 19:09:50.394102 1480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425408bdcf26471ebc43545dabdb9986. Latest consensus state: current_term: 1 leader_uuid: "425408bdcf26471ebc43545dabdb9986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425408bdcf26471ebc43545dabdb9986" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42383 } } }
I20260925 19:09:50.394299 1479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:50.394321 1480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:50.394654 1487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:50.395167 1487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:50.397357 1487 catalog_manager.cc:1347] Generated new cluster ID: 70db55cb5cc34d4d9103125adf8157a5
I20260925 19:09:50.397411 1487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:50.403795 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.392423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.27.174.126:42383)
0925 19:09:50.392663 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.392677 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.392701 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:09:50.392789 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.392794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.392835 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.392963 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.393806 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.395495 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.396607 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.397090 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.397244 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.399814 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.399818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.399818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.399829 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.402182 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.402545 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.402548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.402548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.402645 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.402819 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.402819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.402820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.403083 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.403203 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.403432 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.403563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":136,"threads_started":1}
I20260925 19:09:50.423125 1487 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:50.423894 1487 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:50.430621 1487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 425408bdcf26471ebc43545dabdb9986: Generated new TSK 0
I20260925 19:09:50.430814 1487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:09:50.457873 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:50.537880 1499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:50.538115 1499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:50.538162 1499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:50.539332 1499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:50.539405 1499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:50.539448 1499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:50.540885 1499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:50.541055 1499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:50.541234 1499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:50.545972 1504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:50.546005 1499 server_base.cc:1034] running on GCE node
W20260925 19:09:50.546042 1505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.545979 1507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:50.546340 1499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:50.546586 1499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:50.547748 1499 hybrid_clock.cc:648] HybridClock initialized: now 1790363390547727 us; error 37 us; skew 500 ppm
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.550112 1499 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:09:50.550519 1499 webserver.cc:460] Webserver started at http://127.27.174.65:40631/ using document root <none> and password file <none>
I20260925 19:09:50.550755 1499 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:50.550822 1499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:50.550968 1499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:50.552040 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "bf4624817d224a9c9785a0c94ae65b0c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "6aa3cf6dad5984f4b705d2880822d4c6"
server_key_iv: "72d5d62927c3357ef806bc2d66bc2631"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.552398 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "bf4624817d224a9c9785a0c94ae65b0c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "6aa3cf6dad5984f4b705d2880822d4c6"
server_key_iv: "72d5d62927c3357ef806bc2d66bc2631"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.553697 1499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:50.554471 1514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.554694 1499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:50.554788 1499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "bf4624817d224a9c9785a0c94ae65b0c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "6aa3cf6dad5984f4b705d2880822d4c6"
server_key_iv: "72d5d62927c3357ef806bc2d66bc2631"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.554860 1499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:50.565125 1499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:50.566812 1499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:50.566969 1499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:50.567181 1499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:50.567469 1499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:50.567528 1499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.567566 1499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:50.567593 1499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.573320 1499 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:43419
I20260925 19:09:50.573392 1627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:43419 every 8 connection(s)
I20260925 19:09:50.573688 1499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:50.582855 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.574358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43435 (local address 127.27.174.126:42383)
0925 19:09:50.574402 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.574404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.574662 (+ 258us) server_negotiation.cc:407] Connection header received
0925 19:09:50.575631 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.575632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.575645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.575709 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.576306 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.576814 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.577801 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.578017 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.578104 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.580253 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.580254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.580254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.580265 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.581479 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.581900 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.581901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.581902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.581959 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.582221 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.582222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.582223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.582335 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.582411 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.582679 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.582768 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:50.583034 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.574329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42383 (local address 127.27.174.65:43435)
0925 19:09:50.574503 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:50.574570 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:50.575486 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:50.575774 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:50.575777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:50.576028 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:50.576209 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.576212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.576908 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.576909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.577700 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.577702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.578112 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.578113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.578243 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.578732 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:50.578740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:50.580123 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:50.581578 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.581584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.581586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.581812 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.582025 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.582026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.582027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.582136 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.582401 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:50.582407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:50.582549 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 19:09:50.582658 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:09:50.582809 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260925 19:09:50.583148 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1499
I20260925 19:09:50.583262 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:50.583499 28345 external_mini_cluster.cc:1428] Setting key 4089e5478773aede9d2ff8a22208feec
I20260925 19:09:50.583630 1628 heartbeater.cc:347] Connected to a master server at 127.27.174.126:42383
I20260925 19:09:50.583731 1628 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:50.583912 1628 heartbeater.cc:511] Master 127.27.174.126:42383 requested a full tablet report, sending...
I20260925 19:09:50.584503 1437 ts_manager.cc:194] Registered new tserver with Master: bf4624817d224a9c9785a0c94ae65b0c (127.27.174.65:43419)
I20260925 19:09:50.585384 1437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:43435
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:09:50.607165 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:50.686236 1633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:50.686473 1633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:50.686509 1633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:50.687660 1633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:50.687736 1633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:50.687778 1633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:50.689247 1633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:50.689500 1633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:50.689719 1633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:50.694738 1638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.694761 1641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.694799 1639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:50.694938 1633 server_base.cc:1034] running on GCE node
I20260925 19:09:50.695223 1633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:50.695452 1633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:50.696666 1633 hybrid_clock.cc:648] HybridClock initialized: now 1790363390696652 us; error 27 us; skew 500 ppm
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.698987 1633 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:09:50.699342 1633 webserver.cc:460] Webserver started at http://127.27.174.66:44391/ using document root <none> and password file <none>
I20260925 19:09:50.699507 1633 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:50.699545 1633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:50.699700 1633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:50.700654 1633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "e4df979dcd2a4ccba3aaa08c070a083c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "fd9438a9b41c2cd6fedfa14fca0d6f02"
server_key_iv: "73f5a01a03726340681edd0980dfa31b"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.700942 1633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "e4df979dcd2a4ccba3aaa08c070a083c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "fd9438a9b41c2cd6fedfa14fca0d6f02"
server_key_iv: "73f5a01a03726340681edd0980dfa31b"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.702214 1633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:50.702888 1648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.703056 1633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:50.703137 1633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "e4df979dcd2a4ccba3aaa08c070a083c"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "fd9438a9b41c2cd6fedfa14fca0d6f02"
server_key_iv: "73f5a01a03726340681edd0980dfa31b"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.703188 1633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:50.715193 1633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:50.717120 1633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:50.717224 1633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:50.717402 1633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:50.717636 1633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:50.717664 1633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.717684 1633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:50.717697 1633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.723191 1633 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:43819
I20260925 19:09:50.723263 1761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:43819 every 8 connection(s)
I20260925 19:09:50.723562 1633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:50.731562 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1633
I20260925 19:09:50.731693 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:50.731904 28345 external_mini_cluster.cc:1428] Setting key d7be12839e3606fcd4f58b65e0274528
I20260925 19:09:50.732621 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.724288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37389 (local address 127.27.174.126:42383)
0925 19:09:50.724330 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.724332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.724539 (+ 207us) server_negotiation.cc:407] Connection header received
0925 19:09:50.725478 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.725479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.725491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.725544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.726153 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.726650 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.727641 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.727856 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.727944 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.729988 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.729990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.729990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.729998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.731123 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.731496 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.731497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.731498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.731556 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.731954 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.731955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.731955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.732073 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.732150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.732427 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.732528 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:50.732795 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.724264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42383 (local address 127.27.174.66:37389)
0925 19:09:50.724413 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:50.724463 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:50.725367 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:50.725618 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:50.725621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:50.725856 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:50.726041 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.726045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.726749 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.726750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.727518 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.727520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.727946 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.727947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.728064 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.728563 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:50.728571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:50.729892 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:50.731204 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.731210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.731212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.731420 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.731739 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.731740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.731740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.731885 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.732141 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:50.732146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:50.732293 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 19:09:50.732400 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 19:09:50.732559 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 19:09:50.733367 1762 heartbeater.cc:347] Connected to a master server at 127.27.174.126:42383
I20260925 19:09:50.733453 1762 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:50.733620 1762 heartbeater.cc:511] Master 127.27.174.126:42383 requested a full tablet report, sending...
I20260925 19:09:50.734031 1437 ts_manager.cc:194] Registered new tserver with Master: e4df979dcd2a4ccba3aaa08c070a083c (127.27.174.66:43819)
I20260925 19:09:50.734541 1437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:37389
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:09:50.758191 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:50.836902 1767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:50.837095 1767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:50.837118 1767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:50.838256 1767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:50.838296 1767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:50.838322 1767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:50.839810 1767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:50.840044 1767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:50.840319 1767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:50.845299 1773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.845301 1775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:50.845368 1772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:50.845469 1767 server_base.cc:1034] running on GCE node
I20260925 19:09:50.845762 1767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:50.845943 1767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:50.847129 1767 hybrid_clock.cc:648] HybridClock initialized: now 1790363390847120 us; error 26 us; skew 500 ppm
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.849591 1767 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:09:50.850052 1767 webserver.cc:460] Webserver started at http://127.27.174.67:45945/ using document root <none> and password file <none>
I20260925 19:09:50.850281 1767 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:50.850346 1767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:50.850477 1767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:50.851722 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "5e0dd723195b4a0382af2971be8e61db"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "57b49537759599c5485a8d027b1761af"
server_key_iv: "7c2d442245768d6482435fd5034c3705"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.852125 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "5e0dd723195b4a0382af2971be8e61db"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "57b49537759599c5485a8d027b1761af"
server_key_iv: "7c2d442245768d6482435fd5034c3705"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.853679 1767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:50.854380 1782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.854542 1767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:50.854638 1767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "5e0dd723195b4a0382af2971be8e61db"
format_stamp: "Formatted at 2026-09-25 19:09:50 on dist-test-slave-fcdn"
server_key: "57b49537759599c5485a8d027b1761af"
server_key_iv: "7c2d442245768d6482435fd5034c3705"
server_key_version: "encryptionkey@0"
I20260925 19:09:50.854696 1767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:50.875411 1767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:50.877734 1767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:50.877846 1767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:50.878021 1767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:50.878335 1767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:50.878369 1767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.878415 1767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:50.878443 1767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:50.884246 1767 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:39199
I20260925 19:09:50.884310 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:39199 every 8 connection(s)
I20260925 19:09:50.884629 1767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:09:50.892760 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1767
I20260925 19:09:50.892855 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:50.893033 28345 external_mini_cluster.cc:1428] Setting key 7d9ebf1d5fbfb3ef6270a728513d4b85
I20260925 19:09:50.894133 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.885365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34087 (local address 127.27.174.126:42383)
0925 19:09:50.885412 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.885414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.885617 (+ 203us) server_negotiation.cc:407] Connection header received
0925 19:09:50.886512 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.886513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.886522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.886575 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.887168 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.887658 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.888700 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.888901 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.888961 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.891016 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.891018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.891018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.891026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.892396 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.892969 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.892970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.892971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.893035 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.893365 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.893366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.893367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.893535 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.893600 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.893950 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.894036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:09:50.894371 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.885322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42383 (local address 127.27.174.67:34087)
0925 19:09:50.885483 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:50.885524 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:50.886428 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:50.886641 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:50.886645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:50.886887 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:50.887071 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.887075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.887753 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.887754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.888608 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:50.888611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.889016 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.889018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:50.889134 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.889593 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:50.889601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:50.890913 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:50.892492 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.892499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.892501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.892842 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.893133 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:50.893133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:50.893134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:50.893308 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:50.893592 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:50.893598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:50.893746 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 19:09:50.893925 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 19:09:50.894152 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260925 19:09:50.894886 1896 heartbeater.cc:347] Connected to a master server at 127.27.174.126:42383
I20260925 19:09:50.894999 1896 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:50.895164 1896 heartbeater.cc:511] Master 127.27.174.126:42383 requested a full tablet report, sending...
I20260925 19:09:50.895548 1437 ts_manager.cc:194] Registered new tserver with Master: 5e0dd723195b4a0382af2971be8e61db (127.27.174.67:39199)
I20260925 19:09:50.895994 1437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:34087
I20260925 19:09:50.904812 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/test-admin.keytab.
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.920066 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 19:09:50.921675 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:50.920980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.27.174.65:43419)
0925 19:09:50.921148 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.921153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.921163 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:50.921188 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.921189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.921203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.921268 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.921558 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40760: BlockingRecv error: recv got EOF from 127.0.0.1:40760 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260925 19:09:50.921814 28345 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc)
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.922752 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.923834 28345 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:50.931655 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.924345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.27.174.65:43419)
0925 19:09:50.924400 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.924402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.924456 (+ 54us) server_negotiation.cc:407] Connection header received
0925 19:09:50.924499 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.924499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.924508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.924560 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.924931 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.925685 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.926388 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.926720 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.926783 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.928822 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.928833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.928833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.928844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.930395 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.930788 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.930792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.930792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.930853 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.931080 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.931081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.931081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.931211 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.931367 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.931438 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.931512 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 19:09:50.933338 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:50.932832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.27.174.65:43419)
0925 19:09:50.932869 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.932870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.932928 (+ 58us) server_negotiation.cc:407] Connection header received
0925 19:09:50.932985 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.932986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.932994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.933036 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.933298 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40772: BlockingRecv error: recv got EOF from 127.0.0.1:40772 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:09:50.933403 28345 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.934294 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.935105 28345 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:50.941720 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.935576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.27.174.65:43419)
0925 19:09:50.935656 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.935657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.935759 (+ 102us) server_negotiation.cc:407] Connection header received
0925 19:09:50.935782 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.935783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.935791 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.935835 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.936127 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.936576 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.937228 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.937420 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.937481 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.939274 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.939275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.939275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.939280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.940843 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.941148 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.941148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.941149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.941193 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.941352 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.941353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.941353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.941432 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.941496 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.941613 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.941666 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 19:09:50.943146 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:50.942751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.27.174.65:43419)
0925 19:09:50.942802 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.942804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.942844 (+ 40us) server_negotiation.cc:407] Connection header received
0925 19:09:50.942892 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.942893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.942899 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.942941 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.943118 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40782: BlockingRecv error: recv got EOF from 127.0.0.1:40782 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 19:09:50.943377 28345 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363366557183-28345-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.944307 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:50.945159 28345 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:09:50 dist-test-slave-fcdn krb5kdc[1392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:09:50.951798 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:50.945656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.27.174.65:43419)
0925 19:09:50.945706 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:50.945708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:50.945793 (+ 85us) server_negotiation.cc:407] Connection header received
0925 19:09:50.945834 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:50.945835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:50.945841 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:50.945887 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:50.946207 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.946636 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.947272 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:50.947437 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:50.947482 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:50.949457 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:50.949458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:50.949458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:50.949463 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:50.950763 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.951094 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.951094 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.951095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.951139 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:50.951343 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:50.951344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:50.951344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:50.951423 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:50.951501 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:50.951589 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 19:09:50.951745 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:09:50.952512 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1499
I20260925 19:09:50.957274 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1633
I20260925 19:09:50.962150 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1767
I20260925 19:09:50.966256 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1405
2026-09-25T19:09:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1860 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:51 dist-test-slave-fcdn krb5kdc[1916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:51 dist-test-slave-fcdn krb5kdc[1916](info): set up 2 sockets
Sep 25 19:09:51 dist-test-slave-fcdn krb5kdc[1916](info): commencing operation
krb5kdc: starting...
W20260925 19:09:51.991839 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
I20260925 19:09:52.110286 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44679
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:52.190205 1929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:52.190438 1929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:52.190461 1929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:52.191669 1929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:52.191711 1929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:52.191725 1929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:52.191737 1929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:52.191789 1929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:52.193238 1929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:52.193488 1929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:52.193714 1929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:09:52.198712 1929 server_base.cc:1034] running on GCE node
W20260925 19:09:52.198678 1937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.198698 1934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.198733 1935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.199100 1929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.199384 1929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.200573 1929 hybrid_clock.cc:648] HybridClock initialized: now 1790363392200536 us; error 36 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.202961 1929 init.cc:377] Logged in from keytab as oryx/127.27.174.126@KRBTEST.COM (short username oryx)
I20260925 19:09:52.203444 1929 webserver.cc:460] Webserver started at http://127.27.174.126:33413/ using document root <none> and password file <none>
I20260925 19:09:52.203747 1929 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.203817 1929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.203965 1929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:52.205065 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "2c5633ebab9347208c5c00659f62dd50"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "9c350626ff089a9b082785adfb6a8cbf"
server_key_iv: "cfb630d123c36b235ab12e43707f9126"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.205441 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "2c5633ebab9347208c5c00659f62dd50"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "9c350626ff089a9b082785adfb6a8cbf"
server_key_iv: "cfb630d123c36b235ab12e43707f9126"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.206801 1929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:52.207504 1944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.207697 1929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.207794 1929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "2c5633ebab9347208c5c00659f62dd50"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "9c350626ff089a9b082785adfb6a8cbf"
server_key_iv: "cfb630d123c36b235ab12e43707f9126"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.207883 1929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.226433 1929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:52.228398 1929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:52.228562 1929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:52.232498 1929 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44679
I20260925 19:09:52.232532 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44679 every 8 connection(s)
I20260925 19:09:52.232957 1929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:52.233913 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:52.234823 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 1929
I20260925 19:09:52.234916 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:52.235126 28345 external_mini_cluster.cc:1428] Setting key b61f2c0cd522b0b1220daf87d140a695
I20260925 19:09:52.237280 1997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Bootstrap starting.
I20260925 19:09:52.237850 1997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:52.238148 1997 log.cc:824] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:52.238857 1997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: No bootstrap required, opened a new log
I20260925 19:09:52.241369 1997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } }
I20260925 19:09:52.241521 1997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:52.241545 1997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5633ebab9347208c5c00659f62dd50, State: Initialized, Role: FOLLOWER
I20260925 19:09:52.241673 1997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } }
I20260925 19:09:52.241762 1997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:52.241786 1997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:52.241850 1997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:52.242460 1997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } }
I20260925 19:09:52.242569 1997 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5633ebab9347208c5c00659f62dd50; no voters:
I20260925 19:09:52.242729 1997 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:52.242810 2002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:52.242940 2002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 LEADER]: Becoming Leader. State: Replica: 2c5633ebab9347208c5c00659f62dd50, State: Running, Role: LEADER
I20260925 19:09:52.243031 2002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } }
I20260925 19:09:52.243033 1997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:52.244189 2003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } }
I20260925 19:09:52.244298 2003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:52.244621 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5633ebab9347208c5c00659f62dd50. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } }
I20260925 19:09:52.244779 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:52.244961 2018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:52.245010 2018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:52.245056 2016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:52.245219 2016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:52.246502 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.235658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.27.174.126:44679)
0925 19:09:52.235824 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.235833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.235864 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:09:52.236268 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.236275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.236320 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.236433 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.237184 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.238174 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.239189 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.239648 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.239739 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.242463 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.242467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.242467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.242479 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.244996 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.245430 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.245432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.245432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.245511 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.245718 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.245719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.245719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.245928 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.246064 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.246207 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.246339 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260925 19:09:52.247987 2016 catalog_manager.cc:1347] Generated new cluster ID: 1104f78ee60c4efaa35192681f7ad6b4
I20260925 19:09:52.248041 2016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:52.257459 2016 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:52.257980 2016 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:52.262197 2016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Generated new TSK 0
I20260925 19:09:52.262377 2016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
I20260925 19:09:52.290843 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:52.369904 2023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:52.370095 2023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:52.370118 2023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:52.371271 2023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:52.371318 2023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:52.371344 2023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:52.372881 2023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:52.373112 2023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:52.373356 2023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:52.378288 2031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.378268 2029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.378268 2028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.378505 2023 server_base.cc:1034] running on GCE node
I20260925 19:09:52.378754 2023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.378970 2023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.380103 2023 hybrid_clock.cc:648] HybridClock initialized: now 1790363392380095 us; error 29 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.382527 2023 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20260925 19:09:52.382957 2023 webserver.cc:460] Webserver started at http://127.27.174.65:35343/ using document root <none> and password file <none>
I20260925 19:09:52.383147 2023 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.383186 2023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.383271 2023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:52.384317 2023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "832df0ab2375467da11a100246364c39"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "0daef31865a140b308722ae81449a726"
server_key_iv: "bd45b9f9e17eafbd433788d49bc6866f"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.384626 2023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "832df0ab2375467da11a100246364c39"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "0daef31865a140b308722ae81449a726"
server_key_iv: "bd45b9f9e17eafbd433788d49bc6866f"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.385830 2023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:52.386459 2038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.386680 2023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.386749 2023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "832df0ab2375467da11a100246364c39"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "0daef31865a140b308722ae81449a726"
server_key_iv: "bd45b9f9e17eafbd433788d49bc6866f"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.386797 2023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.399842 2023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:52.402043 2023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:52.402148 2023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:52.402324 2023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:52.402563 2023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:52.402602 2023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.402623 2023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:52.402635 2023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.408255 2023 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38289
I20260925 19:09:52.408296 2151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38289 every 8 connection(s)
I20260925 19:09:52.408599 2023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:52.415416 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2023
I20260925 19:09:52.415526 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:52.415772 28345 external_mini_cluster.cc:1428] Setting key 2784d9324f8b6a99225800c23e638d0c
I20260925 19:09:52.418470 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.409258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:54425)
0925 19:09:52.409429 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.409470 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.409938 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.410211 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.410214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.410861 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.411069 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.411074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.411990 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.411991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.412777 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.412779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.413158 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.413159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.413264 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.413737 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.413745 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.415134 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.416956 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.416967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.416969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.417299 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.417501 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.417501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.417502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.417645 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.417912 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.417919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.418099 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.418196 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.418315 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260925 19:09:52.418468 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.409313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:54425 (local address 127.27.174.126:44679)
0925 19:09:52.409370 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.409372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.409636 (+ 264us) server_negotiation.cc:407] Connection header received
0925 19:09:52.410059 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.410060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.410071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.410136 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.411169 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.411903 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.412856 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.413079 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.413136 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.415245 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.415246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.415246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.415254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.416870 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.417376 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.417376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.417376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.417428 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.417708 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.417708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.417708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.417834 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.417907 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.418249 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.418346 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:09:52.419000 2152 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:52.419111 2152 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:52.419301 2152 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
I20260925 19:09:52.419828 1961 ts_manager.cc:194] Registered new tserver with Master: 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
I20260925 19:09:52.420435 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:54425
WARNING: no policy specified for oryx/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
I20260925 19:09:52.442902 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:52.521832 2157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:52.522022 2157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:52.522043 2157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:52.523192 2157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:52.523234 2157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:52.523260 2157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:52.524715 2157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:52.524917 2157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:52.525130 2157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:52.530246 2162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.530267 2165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.530305 2157 server_base.cc:1034] running on GCE node
W20260925 19:09:52.530246 2163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.530658 2157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.530848 2157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.531989 2157 hybrid_clock.cc:648] HybridClock initialized: now 1790363392531979 us; error 26 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.534461 2157 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20260925 19:09:52.534926 2157 webserver.cc:460] Webserver started at http://127.27.174.66:44595/ using document root <none> and password file <none>
I20260925 19:09:52.535137 2157 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.535178 2157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.535265 2157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:52.536293 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "b6949d1218204e7092d266abbbca6b13"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "2cacf39dea528bdf84c572af1de772f7"
server_key_iv: "0efb03366190fd2a2eec1d6589379e16"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.536579 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "b6949d1218204e7092d266abbbca6b13"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "2cacf39dea528bdf84c572af1de772f7"
server_key_iv: "0efb03366190fd2a2eec1d6589379e16"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.537820 2157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:52.538529 2172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.538725 2157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.538805 2157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "b6949d1218204e7092d266abbbca6b13"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "2cacf39dea528bdf84c572af1de772f7"
server_key_iv: "0efb03366190fd2a2eec1d6589379e16"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.538856 2157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.555397 2157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:52.557715 2157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:52.557825 2157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:52.558007 2157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:52.558319 2157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:52.558354 2157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.558377 2157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:52.558422 2157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.563947 2157 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:36049
I20260925 19:09:52.563995 2285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:36049 every 8 connection(s)
I20260925 19:09:52.564299 2157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:52.567526 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2157
I20260925 19:09:52.567672 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:52.567857 28345 external_mini_cluster.cc:1428] Setting key 0686d9b7c078a1f5aeef588537cd58dd
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:52.574033 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.564959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:44387)
0925 19:09:52.565121 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.565162 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.565649 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.565900 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.565904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.566442 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.566666 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.566671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.567407 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.567407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.568354 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.568357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.568844 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.568845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.568948 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.569459 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.569466 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.570836 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.572716 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.572722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.572724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.572951 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.573155 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.573156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.573156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.573253 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.573552 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.573556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.573668 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.573762 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.573878 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260925 19:09:52.574029 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.564979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44387 (local address 127.27.174.126:44679)
0925 19:09:52.565045 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.565047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.565245 (+ 198us) server_negotiation.cc:407] Connection header received
0925 19:09:52.565756 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.565757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.565767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.565831 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.566760 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.567263 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.568460 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.568765 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.568842 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.570938 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.570939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.570940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.570947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.572636 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.573026 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.573028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.573028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.573088 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.573318 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.573319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.573320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.573474 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.573551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.573805 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.573895 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:52.574589 2286 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:52.574690 2286 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:52.574862 2286 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
I20260925 19:09:52.575235 1961 ts_manager.cc:194] Registered new tserver with Master: b6949d1218204e7092d266abbbca6b13 (127.27.174.66:36049)
I20260925 19:09:52.575804 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:44387
WARNING: no policy specified for oryx/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
I20260925 19:09:52.595149 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:52.673818 2291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:52.674010 2291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:52.674033 2291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:52.675218 2291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:52.675261 2291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:52.675285 2291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:52.676795 2291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:52.677026 2291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:52.677277 2291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:52.682370 2296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.682371 2299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.682457 2297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.682482 2291 server_base.cc:1034] running on GCE node
I20260925 19:09:52.682794 2291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.683025 2291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.684224 2291 hybrid_clock.cc:648] HybridClock initialized: now 1790363392684167 us; error 39 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.686738 2291 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20260925 19:09:52.687165 2291 webserver.cc:460] Webserver started at http://127.27.174.67:41817/ using document root <none> and password file <none>
I20260925 19:09:52.687400 2291 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.687469 2291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.687672 2291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:52.688687 2291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "24c700739ff498189f6221f2f2683c0f"
server_key_iv: "d915493783cda394cbeb439de1eb9b2a"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.689041 2291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "24c700739ff498189f6221f2f2683c0f"
server_key_iv: "d915493783cda394cbeb439de1eb9b2a"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.690424 2291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:52.691133 2306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.691322 2291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.691421 2291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "24c700739ff498189f6221f2f2683c0f"
server_key_iv: "d915493783cda394cbeb439de1eb9b2a"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.691500 2291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.712836 2291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:52.714721 2291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:52.714885 2291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:52.715119 2291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:52.715432 2291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:52.715488 2291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.715529 2291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:52.715557 2291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.721366 2291 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37405
I20260925 19:09:52.721414 2419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37405 every 8 connection(s)
I20260925 19:09:52.721727 2291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:52.729874 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2291
I20260925 19:09:52.729984 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:52.730238 28345 external_mini_cluster.cc:1428] Setting key 0eed2a59b5deb232b5480bd8d8421625
I20260925 19:09:52.731009 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.722364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56455 (local address 127.27.174.126:44679)
0925 19:09:52.722436 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.722439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.722679 (+ 240us) server_negotiation.cc:407] Connection header received
0925 19:09:52.723139 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.723140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.723150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.723203 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.724242 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.724733 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.725730 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.725913 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.725970 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.728195 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.728197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.728197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.728204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.729487 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.729952 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.729953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.729954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.730023 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.730346 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.730347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.730348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.730472 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.730549 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.730841 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.730921 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:09:52.731199 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.722353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:56455)
0925 19:09:52.722553 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.722587 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.723048 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.723283 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.723287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.723934 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.724119 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.724123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.724818 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.724819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.725618 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.725620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.725988 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.725989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.726091 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.726630 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.726640 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.728080 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.729611 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.729617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.729619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.729850 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.730119 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.730120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.730121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.730267 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.730551 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.730558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.730695 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.730805 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.730965 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260925 19:09:52.731666 2420 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:52.731782 2420 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:52.731951 2420 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
I20260925 19:09:52.732348 1961 ts_manager.cc:194] Registered new tserver with Master: dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:52.732734 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:56455
I20260925 19:09:52.742043 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.125.keytab.
Entry for principal oryx/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.125.keytab.
I20260925 19:09:52.766937 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:36157
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.125.keytab
--principal=oryx/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44679,127.27.174.125:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:52.846927 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:52.847157 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:52.847200 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:52.848390 2425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:52.848459 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:52.848482 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:52.848503 2425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:52.848532 2425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:52.849946 2425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44679,127.27.174.125:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.125.keytab
--principal=oryx/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:36157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
W20260925 19:09:52.850090 2425 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.174.126:44679,127.27.174.125:36157'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 19:09:52.850242 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:52.850456 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:52.855221 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.855216 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.855248 2425 server_base.cc:1034] running on GCE node
W20260925 19:09:52.855227 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.855651 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.855966 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.857124 2425 hybrid_clock.cc:648] HybridClock initialized: now 1790363392857113 us; error 25 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.860496 2425 init.cc:377] Logged in from keytab as oryx/127.27.174.125@KRBTEST.COM (short username oryx)
I20260925 19:09:52.860934 2425 webserver.cc:460] Webserver started at http://127.27.174.125:41857/ using document root <none> and password file <none>
I20260925 19:09:52.861166 2425 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.861233 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.861359 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:52.862360 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/instance:
uuid: "281ff51e1aa541ed973c67f62b3415ab"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "658f4937c14eb6cd6dce3484ffb6549f"
server_key_iv: "8b329e5aedb5ccdebd2004dfb5d84a7d"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.862723 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal/instance:
uuid: "281ff51e1aa541ed973c67f62b3415ab"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "658f4937c14eb6cd6dce3484ffb6549f"
server_key_iv: "8b329e5aedb5ccdebd2004dfb5d84a7d"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.864004 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:52.864650 2440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.864813 2425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:52.864905 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "281ff51e1aa541ed973c67f62b3415ab"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "658f4937c14eb6cd6dce3484ffb6549f"
server_key_iv: "8b329e5aedb5ccdebd2004dfb5d84a7d"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.864984 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.880499 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:52.882290 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:52.882434 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:52.886196 2425 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:36157
I20260925 19:09:52.886224 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:36157 every 8 connection(s)
I20260925 19:09:52.886561 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
I20260925 19:09:52.887552 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:52.889146 2493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:52.891438 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2425
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.125@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:52.899058 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.889879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.27.174.126:44679)
0925 19:09:52.889982 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.889984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.890157 (+ 173us) server_negotiation.cc:407] Connection header received
0925 19:09:52.890630 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.890631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.890644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.890696 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.891309 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.892128 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.893408 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.893596 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.893652 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.895936 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.895938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.895938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.895946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.897456 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.897915 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.897916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.897917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.897980 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.898245 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.898247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.898247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.898444 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.898519 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.898864 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.898955 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 19:09:52.899236 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.889844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38728)
0925 19:09:52.890026 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.890071 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.890526 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.890774 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.890777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.891012 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.891202 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.891206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.892216 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.892218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.893331 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.893333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.893669 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.893670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.893799 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.894295 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.894303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.895847 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.897544 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.897550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.897551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.897836 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.898050 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.898051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.898051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.898156 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.898524 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.898529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.898705 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.898833 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.898988 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260925 19:09:52.900007 2493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:52.901283 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.891992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.27.174.125:36157)
0925 19:09:52.892125 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.892134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.892150 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:09:52.892214 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.892216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.892236 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.892327 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.892857 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.893799 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.894879 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.895352 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.895426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.898206 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.898212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.898213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.898219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.900032 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.900437 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.900440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.900440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.900493 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.900713 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.900714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.900714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.900852 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.901008 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.901084 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.901159 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.125@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:52.907044 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.900263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53308)
0925 19:09:52.900446 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.900456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.900504 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.900634 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.900634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.900762 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.900804 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.900805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.901462 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.901463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.902096 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.902099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.902412 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.902413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.902472 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.902871 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.902875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.904077 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.906185 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.906186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.906186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.906356 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.906533 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.906534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.906534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.906611 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.906814 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.906815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.906847 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.906897 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.906966 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260925 19:09:52.907169 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.900324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.27.174.125:36157)
0925 19:09:52.900455 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.900458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.900489 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:09:52.900557 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.900558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.900567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.900631 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.900876 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.901367 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.902167 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.902345 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.902414 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.904181 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.904182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.904182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.904188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.906107 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.906420 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.906421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.906421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.906477 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.906666 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.906667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.906667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.906758 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.906806 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.906911 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.907006 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260925 19:09:52.908422 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Bootstrap starting.
I20260925 19:09:52.908967 2493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:52.909188 2493 log.cc:824] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:52.909880 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: No bootstrap required, opened a new log
I20260925 19:09:52.911367 2493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } }
I20260925 19:09:52.911485 2493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:52.911506 2493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 281ff51e1aa541ed973c67f62b3415ab, State: Initialized, Role: FOLLOWER
I20260925 19:09:52.911646 2493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } }
I20260925 19:09:52.911886 2500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } } }
I20260925 19:09:52.911963 2493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:52.911972 2500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:09:52.913486 2511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:52.913543 2511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:52.915169 2425 master_runner.cc:428] Detected that this master 281ff51e1aa541ed973c67f62b3415ab is joining an existing cluster
I20260925 19:09:52.915227 2425 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.174.125:36157
I20260925 19:09:52.922883 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.916210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53324)
0925 19:09:52.916513 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.916527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.916591 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.916731 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.916731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.916934 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.916987 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.916989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.917901 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.917901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.918428 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.918429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.918945 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.918945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.919001 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.919432 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.919436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.919867 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.921797 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.921798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.921798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.921989 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.922173 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.922173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.922173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.922223 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.922476 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.922477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.922509 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.922601 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.922778 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":80,"threads_started":1}
I20260925 19:09:52.922997 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.916392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.27.174.125:36157)
0925 19:09:52.916449 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.916451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.916581 (+ 130us) server_negotiation.cc:407] Connection header received
0925 19:09:52.916666 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.916666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.916677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.916727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.917064 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.917824 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.918525 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.918856 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.918932 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.919951 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.919953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.919953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.919960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.921710 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.922060 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.922061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.922061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.922110 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.922285 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.922285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.922285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.922396 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.922436 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.922629 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.922858 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":42}
I20260925 19:09:52.923687 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.916199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38744)
0925 19:09:52.916354 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.916365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.916427 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.916611 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.916612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.916895 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.916987 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.916989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.918766 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.918767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.919645 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.919647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.920114 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.920115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.920193 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.920644 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.920647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.920894 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.922529 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.922530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.922530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.922803 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.923020 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.923020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.923021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.923073 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.923319 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.923320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.923373 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.923459 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.923560 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260925 19:09:52.923704 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.916254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.27.174.126:44679)
0925 19:09:52.916299 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.916301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.916417 (+ 116us) server_negotiation.cc:407] Connection header received
0925 19:09:52.916502 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.916503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.916514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.916587 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.917926 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.918672 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.919727 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.920030 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.920114 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.920985 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.920986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.920986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.920994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.922427 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.922893 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.922894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.922894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.922949 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.923139 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.923139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.923140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.923250 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.923328 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.923494 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.923584 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:52.933036 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.925908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53334)
0925 19:09:52.926006 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.926048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.926115 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.926252 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.926254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.926481 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.926524 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.926526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.927396 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.927397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.928071 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.928074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.928517 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.928518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.928572 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.929008 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.929012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.930294 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.931996 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.932002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.932004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.932221 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.932431 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.932432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.932432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.932501 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.932754 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.932759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.932802 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.932865 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.932956 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":51,"threads_started":1}
I20260925 19:09:52.933068 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.925964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.27.174.125:36157)
0925 19:09:52.926002 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.926004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.926114 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:09:52.926182 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.926183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.926192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.926250 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.926600 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.927310 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.928148 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.928441 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.928513 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.930371 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.930372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.930373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.930379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.931900 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.932304 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.932305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.932305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.932361 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.932563 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.932564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.932564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.932693 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.932757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.932909 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.932980 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:52.933672 1961 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.174.125:36157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab"
member_type: NON_VOTER
last_known_addr {
host: "127.27.174.125"
port: 36157
}
attrs {
promote: true
}
}
dest_uuid: "2c5633ebab9347208c5c00659f62dd50"
cas_config_opid_index: -1
I20260925 19:09:52.933987 1961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } }
I20260925 19:09:52.934469 2520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } }
I20260925 19:09:52.934754 2467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:52.935401 2520 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.174.125:36157
I20260925 19:09:52.935555 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } }
I20260925 19:09:52.935634 2425 master.cc:537] Master@127.27.174.125:36157 shutting down...
I20260925 19:09:52.935649 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:52.935693 2467 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 19:09:52.935990 2520 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:09:52.936666 1947 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 19:09:52.937196 1947 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 -> Peer 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125:36157): Couldn't send request to peer 281ff51e1aa541ed973c67f62b3415ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:09:52.938254 2425 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:52.938338 2425 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:52.938385 2425 tablet_replica.cc:330] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: stopping tablet replica
I20260925 19:09:52.950856 2425 master.cc:552] Master@127.27.174.125:36157 shutdown complete.
I20260925 19:09:52.950923 2425 master_runner.cc:305] Clearing existing system tablet
I20260925 19:09:52.951062 2425 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.951128 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.951855 2425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:52.952503 2525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.952658 2425 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:52.952715 2425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:52.952739 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "281ff51e1aa541ed973c67f62b3415ab"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "658f4937c14eb6cd6dce3484ffb6549f"
server_key_iv: "8b329e5aedb5ccdebd2004dfb5d84a7d"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.953334 2425 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:52.954696 2425 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:09:52.954751 2425 log.cc:1196] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 19:09:52.954947 2425 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Deleting consensus metadata
I20260925 19:09:52.955068 2425 master_runner.cc:315] Copying system tablet from 127.27.174.126:44679
I20260925 19:09:52.955807 2425 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.174.126:44679
I20260925 19:09:52.962369 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.956381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38746)
0925 19:09:52.956522 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:52.956565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:52.956646 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:52.956809 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:52.956809 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:52.956959 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:52.957014 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.957015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.957952 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.957953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.958495 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:52.958497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.958963 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.958964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:52.959012 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.959482 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:52.959487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:52.959857 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:52.961479 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.961480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.961480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.961635 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.961827 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:52.961828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:52.961828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:52.961876 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:52.962133 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:52.962134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:52.962167 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 19:09:52.962225 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:09:52.962295 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 19:09:52.962425 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:52.956545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.27.174.126:44679)
0925 19:09:52.956596 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:52.956598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:52.956612 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:09:52.956704 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:52.956705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:52.956715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:52.956793 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:52.957084 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.957872 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.958581 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:52.958891 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:52.958961 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:52.959932 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:52.959933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:52.959933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:52.959940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:52.961401 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.961703 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.961704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.961705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.961763 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:52.961934 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:52.961935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:52.961935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:52.962064 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:52.962133 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:52.962261 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 19:09:52.962335 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:52.962726 1981 tablet_copy_service.cc:140] P 2c5633ebab9347208c5c00659f62dd50: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 281ff51e1aa541ed973c67f62b3415ab ({username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:38746)
I20260925 19:09:52.962822 1981 tablet_copy_service.cc:161] P 2c5633ebab9347208c5c00659f62dd50: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 281ff51e1aa541ed973c67f62b3415ab at {username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:38746: session id = 281ff51e1aa541ed973c67f62b3415ab-00000000000000000000000000000000
I20260925 19:09:52.963404 1981 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:09:52.964033 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:09:52.964875 2425 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: tablet copy: Starting download of 0 data blocks...
I20260925 19:09:52.965014 2425 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: tablet copy: Starting download of 1 WAL segments...
I20260925 19:09:52.965808 2425 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:09:52.966548 1981 tablet_copy_service.cc:342] P 2c5633ebab9347208c5c00659f62dd50: Request end of tablet copy session 281ff51e1aa541ed973c67f62b3415ab-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:38746
I20260925 19:09:52.966604 1981 tablet_copy_service.cc:433] P 2c5633ebab9347208c5c00659f62dd50: ending tablet copy session 281ff51e1aa541ed973c67f62b3415ab-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 281ff51e1aa541ed973c67f62b3415ab
I20260925 19:09:52.968830 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:52.969925 2532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.969937 2533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:52.970117 2535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:52.970256 2425 server_base.cc:1034] running on GCE node
I20260925 19:09:52.970347 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:52.970491 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:52.971658 2425 hybrid_clock.cc:648] HybridClock initialized: now 1790363392971651 us; error 22 us; skew 500 ppm
Sep 25 19:09:52 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:52.973083 2425 init.cc:377] Logged in from keytab as oryx/127.27.174.125@KRBTEST.COM (short username oryx)
I20260925 19:09:52.973353 2425 webserver.cc:460] Webserver started at http://127.27.174.125:37795/ using document root <none> and password file <none>
I20260925 19:09:52.973470 2425 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:52.973517 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:52.974089 2425 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.974587 2542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:52.974716 2425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:52.974772 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "281ff51e1aa541ed973c67f62b3415ab"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "658f4937c14eb6cd6dce3484ffb6549f"
server_key_iv: "8b329e5aedb5ccdebd2004dfb5d84a7d"
server_key_version: "encryptionkey@0"
I20260925 19:09:52.974826 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.007666 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.009478 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.012553 2425 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:36157
I20260925 19:09:53.012579 2594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:36157 every 8 connection(s)
I20260925 19:09:53.012897 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
I20260925 19:09:53.013548 2595 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:09:53.013993 2595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Bootstrap starting.
I20260925 19:09:53.016294 2595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:09:53.016516 2595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Bootstrap complete.
I20260925 19:09:53.016716 2595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } }
I20260925 19:09:53.016781 2595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 281ff51e1aa541ed973c67f62b3415ab, State: Initialized, Role: LEARNER
I20260925 19:09:53.016806 2595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } }
I20260925 19:09:53.016991 2595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:53.017001 2598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } }
I20260925 19:09:53.017113 2598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: LEARNER
I20260925 19:09:53.018378 2609 catalog_manager.cc:1259] Loaded cluster ID: 1104f78ee60c4efaa35192681f7ad6b4
I20260925 19:09:53.018435 2609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: loading cluster ID for follower catalog manager: success
I20260925 19:09:53.019318 2609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: acquiring CA information for follower catalog manager: success
I20260925 19:09:53.019583 2609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:09:53.063721 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.053417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.27.174.125:36157)
0925 19:09:53.053578 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.053580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.053586 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:09:53.053609 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.053610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.053619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.053702 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.054269 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.054958 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.055727 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.055935 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.056016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.061136 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.061138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.061138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.061152 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.062536 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.062913 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.062914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.062914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.062975 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.063176 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.063177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.063177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.063295 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.063397 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.063502 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.063587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.27.174.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.124.keytab.
Entry for principal oryx/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.124.keytab.
I20260925 19:09:53.088028 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:46095
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.124.keytab
--principal=oryx/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:53.167869 2613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:53.168087 2613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:53.168124 2613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:53.169252 2613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:53.169317 2613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:53.169340 2613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:53.169373 2613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:53.169400 2613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:53.170842 2613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.124.keytab
--principal=oryx/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:53.171101 2613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:53.171327 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:09:53.176299 2613 server_base.cc:1034] running on GCE node
W20260925 19:09:53.176312 2619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.176330 2621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.176312 2618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.176772 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:53.177095 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:53.178256 2613 hybrid_clock.cc:648] HybridClock initialized: now 1790363393178250 us; error 40 us; skew 500 ppm
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:53.180651 2613 init.cc:377] Logged in from keytab as oryx/127.27.174.124@KRBTEST.COM (short username oryx)
I20260925 19:09:53.181046 2613 webserver.cc:460] Webserver started at http://127.27.174.124:37561/ using document root <none> and password file <none>
I20260925 19:09:53.181277 2613 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.181342 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.181484 2613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:53.182476 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/instance:
uuid: "2e1b8769edef4147b426d62574254220"
format_stamp: "Formatted at 2026-09-25 19:09:53 on dist-test-slave-fcdn"
server_key: "ef140442767eeb6b5abb18181157d93b"
server_key_iv: "64ff237dee838b28c48d98ed87b6ee50"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.182818 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal/instance:
uuid: "2e1b8769edef4147b426d62574254220"
format_stamp: "Formatted at 2026-09-25 19:09:53 on dist-test-slave-fcdn"
server_key: "ef140442767eeb6b5abb18181157d93b"
server_key_iv: "64ff237dee838b28c48d98ed87b6ee50"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.184175 2613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:53.184864 2628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.185022 2613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:53.185118 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "2e1b8769edef4147b426d62574254220"
format_stamp: "Formatted at 2026-09-25 19:09:53 on dist-test-slave-fcdn"
server_key: "ef140442767eeb6b5abb18181157d93b"
server_key_iv: "64ff237dee838b28c48d98ed87b6ee50"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.185194 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.207810 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.209714 2613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:53.209883 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.214190 2613 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:46095
I20260925 19:09:53.214233 2680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:46095 every 8 connection(s)
I20260925 19:09:53.214610 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
I20260925 19:09:53.215559 2681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:53.217170 2681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:53.222332 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2613
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.124@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:53.226557 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.218003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.27.174.126:44679)
0925 19:09:53.218072 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.218074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.218135 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:09:53.218610 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.218611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.218624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.218685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.219295 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.219862 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.220849 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.221040 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.221117 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.223478 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.223479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.223480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.223486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.225122 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.225578 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.225579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.225579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.225663 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.225921 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.225921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.225922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.226052 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.226121 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.226352 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.226440 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:09:53.226640 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.217843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38752)
0925 19:09:53.218006 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.218046 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.218496 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.218759 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.218762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.219001 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.219194 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.219197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.219951 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.219952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.220737 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.220739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.221135 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.221136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.221261 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.221752 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.221760 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.223393 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.225197 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.225203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.225205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.225501 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.225725 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.225725 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.225726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.225854 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.226120 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.226125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.226227 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.226310 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.226419 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":108,"threads_started":1}
I20260925 19:09:53.227428 2681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.124@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:53.232609 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.222815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.27.174.124:46095)
0925 19:09:53.222982 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.222987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.223011 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:09:53.223037 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.223038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.223053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.223121 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.223904 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.225029 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.226030 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.226460 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.226528 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.229270 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.229276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.229276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.229282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.231188 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.231700 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.231703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.231703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.231759 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.231949 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.231949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.231950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.232100 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.232258 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.232345 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.232439 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260925 19:09:53.236208 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.227720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53374)
0925 19:09:53.227763 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.227773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.227806 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.227985 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.227986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.228147 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.228199 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.228201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.229040 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.229041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.229628 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.229630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.230061 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.230062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.230157 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.230479 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.230485 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.231933 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.235450 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.235451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.235451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.235640 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.235802 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.235803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.235803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.235850 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.236003 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.236003 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.236030 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.236079 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.236153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:09:53.236209 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.227805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.27.174.125:36157)
0925 19:09:53.227871 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.227873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.227880 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:53.227902 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.227903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.227915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.227983 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.228303 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.228926 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.229716 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.229991 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.230045 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.232003 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.232004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.232004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.232010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.235383 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.235707 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.235708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.235708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.235752 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.235884 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.235885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.235885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.235969 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.236011 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.236095 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.236152 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:09:53.236591 2681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.124@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
I20260925 19:09:53.242569 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.236874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:52990)
0925 19:09:53.236922 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.236932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.236971 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.237089 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.237089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.237193 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.237216 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.237217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.237736 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.237737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.238183 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.238184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.238494 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.238495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.238535 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.238836 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.238839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.239983 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.241575 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.241575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.241576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.241738 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.241960 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.241961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.241961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.242012 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.242263 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.242264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.242298 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.242381 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.242469 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:09:53.242628 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.236913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.27.174.124:46095)
0925 19:09:53.236950 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.236952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.236960 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:53.237026 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.237027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.237034 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.237084 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.237256 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.237680 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.238257 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.238435 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.238487 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.240081 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.240082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.240082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.240088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.241476 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.241832 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.241833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.241833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.241884 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.242091 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.242091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.242091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.242194 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.242260 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.242403 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.242516 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":16}
I20260925 19:09:53.243929 2681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Bootstrap starting.
I20260925 19:09:53.244711 2681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:53.244973 2681 log.cc:824] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:53.245601 2681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: No bootstrap required, opened a new log
I20260925 19:09:53.247015 2681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } }
I20260925 19:09:53.247138 2681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:53.247191 2681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1b8769edef4147b426d62574254220, State: Initialized, Role: FOLLOWER
I20260925 19:09:53.247318 2681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } }
I20260925 19:09:53.247592 2681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:53.247560 2686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } } }
I20260925 19:09:53.247740 2686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:09:53.249272 2697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:53.249348 2697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:53.251504 2613 master_runner.cc:428] Detected that this master 2e1b8769edef4147b426d62574254220 is joining an existing cluster
I20260925 19:09:53.251554 2613 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.174.124:46095
I20260925 19:09:53.279296 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.27.174.124:46095)
0925 19:09:53.252912 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.252914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.252959 (+ 45us) server_negotiation.cc:407] Connection header received
0925 19:09:53.253027 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.253028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.253037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.253093 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.253587 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254296 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255323 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255587 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255657 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.276487 (+ 20830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.276489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.276489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.276496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.278080 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278431 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.278432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.278433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.278474 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278650 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.278651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.278651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.278773 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.278810 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.279115 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.279194 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:09:53.279748 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.27.174.125:36157)
0925 19:09:53.252807 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.252809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.253106 (+ 297us) server_negotiation.cc:407] Connection header received
0925 19:09:53.253197 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.253198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.253208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.253237 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.253647 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254365 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255463 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255785 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255822 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.276534 (+ 20712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.276536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.276536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.276543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.278361 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278809 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.278810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.278810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.278868 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.279074 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.279075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.279075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.279215 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.279260 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.279527 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.279631 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:53.279745 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:52998)
0925 19:09:53.252910 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.252918 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.252968 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.253129 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.253129 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.253421 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.253504 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.253506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254382 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.254383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.255234 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.255236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255724 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.255784 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.256201 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.256206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.276366 (+ 20160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.278169 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.278171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.278361 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.278534 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278534 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.278534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.278578 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.278848 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.278848 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.278885 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.279500 (+ 615us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.279639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":1,"spinlock_wait_cycles":8576,"thread_start_us":99,"threads_started":1}
I20260925 19:09:53.280239 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53384)
0925 19:09:53.253059 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.253067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.253126 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.253296 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.253296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.253516 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.253569 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.253570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254445 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.254446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.255362 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.255364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255897 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.256221 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.256619 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.256626 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.276407 (+ 19781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.278462 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.278464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.278713 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.278946 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.278947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.278947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.279003 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.279308 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.279309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.279419 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.279890 (+ 471us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.280143 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":322,"thread_start_us":90,"threads_started":1}
I20260925 19:09:53.281188 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.27.174.126:44679)
0925 19:09:53.252775 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.252777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.252941 (+ 164us) server_negotiation.cc:407] Connection header received
0925 19:09:53.253044 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.253045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.253057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.253091 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.253574 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254336 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255463 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255780 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255822 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.277191 (+ 21369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.277194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.277194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.277628 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.279337 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.280413 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.280414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.280415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.280477 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.280684 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.280684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.280685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.280815 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.280857 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.281030 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.281104 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:53.281209 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.252537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38758)
0925 19:09:53.252875 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.252885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.252967 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.253157 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.253158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.253404 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.253480 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.253482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.254433 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.254434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.255362 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.255364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.255898 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.255899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.255965 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.276269 (+ 20304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.276275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.276670 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.279649 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.279650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.279650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.280215 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.280554 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.280554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.280555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.280612 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.280892 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.280892 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.280921 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.280993 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.281096 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":32,"thread_start_us":260,"threads_started":1}
I20260925 19:09:53.284066 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2023
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
I20260925 19:09:53.291074 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.283232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:53004)
0925 19:09:53.283278 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.283288 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.283324 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.283461 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.283461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.283595 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.283713 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.283715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.284595 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.284596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.285524 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.285526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.285994 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.285995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.286070 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.286470 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.286476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.288107 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.289976 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.289977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.289977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.290188 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.290398 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.290399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.290399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.290464 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.290759 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.290760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.290804 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.290878 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.290975 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:09:53.291361 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.283283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.27.174.124:46095)
0925 19:09:53.283321 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.283323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.283332 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:53.283387 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.283388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.283398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.283456 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.283781 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.284485 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.285605 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.285917 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.285985 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.288193 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.288194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.288195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.288202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.289892 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.290265 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.290266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.290266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.290325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.290534 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.290535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.290535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.290686 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.290754 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.290916 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.290994 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:53.292730 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:38289
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:44679
--webserver_port=35343
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:53.292699 1961 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.174.124:46095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2e1b8769edef4147b426d62574254220"
member_type: NON_VOTER
last_known_addr {
host: "127.27.174.124"
port: 46095
}
attrs {
promote: true
}
}
dest_uuid: "2c5633ebab9347208c5c00659f62dd50"
cas_config_opid_index: 5
I20260925 19:09:53.292984 1961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } }
I20260925 19:09:53.293466 2521 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e1b8769edef4147b426d62574254220 (127.27.174.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.294458 2521 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.174.124:46095
I20260925 19:09:53.294554 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.294653 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.294795 2613 master.cc:537] Master@127.27.174.124:46095 shutting down...
I20260925 19:09:53.294945 2655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:53.296398 2655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 19:09:53.296744 2520 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:09:53.297416 1945 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 -> Peer 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095): Couldn't send request to peer 2e1b8769edef4147b426d62574254220. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:09:53.298410 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.293467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.27.174.125:36157)
0925 19:09:53.293507 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.293508 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.294735 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 19:09:53.294763 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.294764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.294787 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.294845 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.295025 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.295579 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.297422 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.298015 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.298087 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.298153 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.298262 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.298311 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:53.298722 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.293422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:53398)
0925 19:09:53.294630 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.294641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.294680 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.294908 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.294909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.294911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.294951 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.294952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.295726 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.295727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.297334 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.297337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.298090 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.298091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.298148 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.298171 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.298596 (+ 425us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.298644 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192}
I20260925 19:09:53.298861 2613 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:53.298949 2613 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:53.299000 2613 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: stopping tablet replica
I20260925 19:09:53.299062 2569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 LEARNER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 19:09:53.299248 2520 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 19:09:53.299790 2569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e1b8769edef4147b426d62574254220 (127.27.174.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.300808 2598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5633ebab9347208c5c00659f62dd50. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.300889 2598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: LEARNER
I20260925 19:09:53.300920 2598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: NON_VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: true } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.300959 2598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: LEARNER
I20260925 19:09:53.301445 2520 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: attempting to promote NON_VOTER 281ff51e1aa541ed973c67f62b3415ab to VOTER
I20260925 19:09:53.301637 2520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } }
I20260925 19:09:53.302050 2569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 LEARNER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 19:09:53.302248 2521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 19:09:53.302856 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.302711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:53006)
0925 19:09:53.302769 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.302819 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:09:53.302991 2521 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.303339 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.303168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:53016)
0925 19:09:53.303279 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.303307 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 19:09:53.303464 1945 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 -> Peer 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095): Couldn't send request to peer 2e1b8769edef4147b426d62574254220. Status: Network error: Client connection negotiation failed: client connection to 127.27.174.124:46095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 19:09:53.303874 2521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.303946 2521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.303912 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.304008 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.304092 2569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.305032 2708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.305127 2708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:53.312183 2613 master.cc:552] Master@127.27.174.124:46095 shutdown complete.
I20260925 19:09:53.312229 2613 master_runner.cc:305] Clearing existing system tablet
I20260925 19:09:53.312353 2613 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.312397 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.313109 2613 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:53.313644 2711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.313755 2613 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.313798 2613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.313818 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "2e1b8769edef4147b426d62574254220"
format_stamp: "Formatted at 2026-09-25 19:09:53 on dist-test-slave-fcdn"
server_key: "ef140442767eeb6b5abb18181157d93b"
server_key_iv: "64ff237dee838b28c48d98ed87b6ee50"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.314576 2613 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:53.316082 2613 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:09:53.316157 2613 log.cc:1196] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 19:09:53.316340 2613 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Deleting consensus metadata
I20260925 19:09:53.316488 2613 master_runner.cc:315] Copying system tablet from 127.27.174.126:44679
I20260925 19:09:53.317061 2613 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: tablet copy: Beginning tablet copy session from remote peer at address 127.27.174.126:44679
I20260925 19:09:53.324167 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.317343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:38770)
0925 19:09:53.317452 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.317463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.317521 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.317677 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.317678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.317892 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.317940 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.317942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.318906 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.318908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.319733 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.319736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.320206 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.320207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.320266 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.320773 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.320778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.321170 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.322996 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.322997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.322997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.323221 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.323437 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.323438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.323438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.323503 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.323843 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.323844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.323890 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.323965 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.324068 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260925 19:09:53.324185 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.317397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.27.174.126:44679)
0925 19:09:53.317437 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.317439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.317526 (+ 87us) server_negotiation.cc:407] Connection header received
0925 19:09:53.317572 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.317573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.317583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.317671 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.318031 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.318823 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.319822 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.320117 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.320192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.321262 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.321264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.321264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.321272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.322903 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.323302 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.323303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.323304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.323365 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.323570 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.323571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.323571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.323769 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.323841 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.324006 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.324081 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:53.324882 1981 tablet_copy_service.cc:140] P 2c5633ebab9347208c5c00659f62dd50: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2e1b8769edef4147b426d62574254220 ({username='oryx', principal='oryx/127.27.174.124@KRBTEST.COM'} at 127.0.0.1:38770)
I20260925 19:09:53.324942 1981 tablet_copy_service.cc:161] P 2c5633ebab9347208c5c00659f62dd50: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2e1b8769edef4147b426d62574254220 at {username='oryx', principal='oryx/127.27.174.124@KRBTEST.COM'} at 127.0.0.1:38770: session id = 2e1b8769edef4147b426d62574254220-00000000000000000000000000000000
I20260925 19:09:53.325510 1981 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:09:53.326001 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:09:53.327174 2613 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: tablet copy: Starting download of 0 data blocks...
I20260925 19:09:53.327316 2613 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: tablet copy: Starting download of 1 WAL segments...
I20260925 19:09:53.328065 2613 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:09:53.329012 1981 tablet_copy_service.cc:342] P 2c5633ebab9347208c5c00659f62dd50: Request end of tablet copy session 2e1b8769edef4147b426d62574254220-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.174.124@KRBTEST.COM'} at 127.0.0.1:38770
I20260925 19:09:53.329066 1981 tablet_copy_service.cc:433] P 2c5633ebab9347208c5c00659f62dd50: ending tablet copy session 2e1b8769edef4147b426d62574254220-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2e1b8769edef4147b426d62574254220
I20260925 19:09:53.331677 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:53.332825 2721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.332998 2719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.333117 2613 server_base.cc:1034] running on GCE node
W20260925 19:09:53.336076 2718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.336264 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:53.336400 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:53.337538 2613 hybrid_clock.cc:648] HybridClock initialized: now 1790363393337527 us; error 25 us; skew 500 ppm
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:53.339141 2613 init.cc:377] Logged in from keytab as oryx/127.27.174.124@KRBTEST.COM (short username oryx)
I20260925 19:09:53.339403 2613 webserver.cc:460] Webserver started at http://127.27.174.124:41137/ using document root <none> and password file <none>
I20260925 19:09:53.339517 2613 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.339545 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.340297 2613 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.340867 2728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.341065 2613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:53.341115 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "2e1b8769edef4147b426d62574254220"
format_stamp: "Formatted at 2026-09-25 19:09:53 on dist-test-slave-fcdn"
server_key: "ef140442767eeb6b5abb18181157d93b"
server_key_iv: "64ff237dee838b28c48d98ed87b6ee50"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.341167 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.372707 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.374801 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.378064 2613 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:46095
I20260925 19:09:53.378118 2780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:46095 every 8 connection(s)
I20260925 19:09:53.378544 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
I20260925 19:09:53.379633 2781 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:09:53.380228 2781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Bootstrap starting.
I20260925 19:09:53.383284 2781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:09:53.383507 2781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: Bootstrap complete.
I20260925 19:09:53.383751 2781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } }
I20260925 19:09:53.383849 2781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e1b8769edef4147b426d62574254220, State: Initialized, Role: LEARNER
I20260925 19:09:53.383888 2781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } }
I20260925 19:09:53.384047 2781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:53.384052 2784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.384308 2784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: LEARNER
I20260925 19:09:53.385577 2795 catalog_manager.cc:1259] Loaded cluster ID: 1104f78ee60c4efaa35192681f7ad6b4
I20260925 19:09:53.385861 2795 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: loading cluster ID for follower catalog manager: success
I20260925 19:09:53.387271 2795 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: acquiring CA information for follower catalog manager: success
I20260925 19:09:53.387565 2795 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 19:09:53.392562 2706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:53.392781 2706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:53.392830 2706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:53.393968 2706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:53.394037 2706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:53.394083 2706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:53.395689 2706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:38289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=35343
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:53.395929 2706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:53.396167 2706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:53.400903 2801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.401052 2803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.400949 2800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.401288 2706 server_base.cc:1034] running on GCE node
I20260925 19:09:53.401505 2706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:53.401750 2706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:53.402922 2706 hybrid_clock.cc:648] HybridClock initialized: now 1790363393402902 us; error 35 us; skew 500 ppm
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:53.405491 2706 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20260925 19:09:53.405954 2706 webserver.cc:460] Webserver started at http://127.27.174.65:35343/ using document root <none> and password file <none>
I20260925 19:09:53.406177 2706 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.406244 2706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.407955 2706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:53.408478 2810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.408666 2706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.408766 2706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "832df0ab2375467da11a100246364c39"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "0daef31865a140b308722ae81449a726"
server_key_iv: "bd45b9f9e17eafbd433788d49bc6866f"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.409103 2706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.419282 2706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.421121 2706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:53.421283 2706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.421504 2706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:53.421799 2706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:53.421855 2706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.421895 2706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:53.421922 2706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.427358 2706 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38289
I20260925 19:09:53.427451 2923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38289 every 8 connection(s)
I20260925 19:09:53.427811 2706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:09:53.433058 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2706
I20260925 19:09:53.433215 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2157
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:53.439469 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39965 (local address 127.27.174.124:46095)
0925 19:09:53.428976 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.428979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.428990 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:09:53.430102 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.430103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.430114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.430193 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.430856 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431652 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.432981 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.433296 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.433731 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.436119 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.436121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.436121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.436140 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.437925 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438393 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.438394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.438394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.438442 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438692 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.438693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.438694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.438813 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.438885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.439194 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.439300 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260925 19:09:53.439514 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35769 (local address 127.27.174.126:44679)
0925 19:09:53.429041 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.429043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.429178 (+ 135us) server_negotiation.cc:407] Connection header received
0925 19:09:53.429968 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.429969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.429982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.430041 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.430863 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431641 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.432987 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.433295 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.433336 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.436469 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.436470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.436471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.436478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.438073 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438565 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.438566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.438567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.438635 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438851 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.438852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.438852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.439004 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.439089 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.439357 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.439432 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:09:53.439559 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.27.174.65:39965)
0925 19:09:53.428821 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.428835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.429781 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.430214 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.430215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.430516 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.430754 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.430758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431813 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.431813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.432885 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.432887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.433390 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.433391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.433520 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.434099 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.434108 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.436014 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.438028 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.438037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.438326 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.438510 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.438511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.438622 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.438890 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.438896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.439030 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.439155 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.439337 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":98,"thread_start_us":97,"threads_started":1}
I20260925 19:09:53.439817 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:35769)
0925 19:09:53.429095 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.429114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.429788 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.430067 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.430071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.430518 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.430754 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.430760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431765 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.431766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.432885 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.432887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.433885 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.433885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.433964 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.434372 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.434377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.436402 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.438179 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.438181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.438471 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.438711 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.438712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.438712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.438781 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.439173 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.439173 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.439210 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.439271 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.439705 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":424,"thread_start_us":67,"threads_started":1}
I20260925 19:09:53.440131 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:36049
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:44679
--webserver_port=44595
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:53.440191 2925 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:53.440280 2925 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.440469 2925 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
I20260925 19:09:53.440917 1961 ts_manager.cc:194] Re-registered known tserver with Master: 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
I20260925 19:09:53.440960 2924 heartbeater.cc:347] Connected to a master server at 127.27.174.124:46095
I20260925 19:09:53.441015 2924 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.441082 2924 heartbeater.cc:511] Master 127.27.174.124:46095 requested a full tablet report, sending...
I20260925 19:09:53.441431 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:35769
I20260925 19:09:53.441464 2740 ts_manager.cc:194] Registered new tserver with Master: 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:53.446065 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:37587 (local address 127.27.174.125:36157)
0925 19:09:53.429003 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.429005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.429017 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:53.429895 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.429896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.429909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.429977 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.430863 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431638 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.432853 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.433193 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.433268 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.443376 (+ 10108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.443377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.443378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.443385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.444952 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.445297 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.445299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.445299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.445351 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.445553 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.445554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.445554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.445680 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.445748 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.445898 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.445974 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:09:53.446084 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.428565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.27.174.65:37587)
0925 19:09:53.428727 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.428768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.429781 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.430067 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.430071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.430523 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.430754 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.430760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.431753 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.431754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.432750 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.432753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.434653 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.434654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.434739 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.439923 (+ 5184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.440016 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.443285 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.445035 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.445037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.445037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.445212 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.445423 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.445424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.445425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.445481 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.445751 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.445753 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.445791 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.445876 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.445979 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":127,"thread_start_us":90,"threads_started":1}
I20260925 19:09:53.446503 2926 heartbeater.cc:347] Connected to a master server at 127.27.174.125:36157
I20260925 19:09:53.446566 2926 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.446645 2926 heartbeater.cc:511] Master 127.27.174.125:36157 requested a full tablet report, sending...
I20260925 19:09:53.447003 2559 ts_manager.cc:194] Registered new tserver with Master: 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
W20260925 19:09:53.524600 2932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:53.524791 2932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:53.524813 2932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:53.525962 2932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:53.526005 2932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:53.526031 2932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:53.527467 2932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:36049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=44595
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:53.527746 2932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:53.527979 2932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:53.532806 2938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.532801 2940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.532829 2937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.533459 2932 server_base.cc:1034] running on GCE node
I20260925 19:09:53.533718 2932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:53.533952 2932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:53.535135 2932 hybrid_clock.cc:648] HybridClock initialized: now 1790363393535122 us; error 49 us; skew 500 ppm
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:53.537776 2932 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20260925 19:09:53.538142 2932 webserver.cc:460] Webserver started at http://127.27.174.66:44595/ using document root <none> and password file <none>
I20260925 19:09:53.538328 2932 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.538378 2932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.539743 2932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:53.540277 2947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.540427 2932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:53.540508 2932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "b6949d1218204e7092d266abbbca6b13"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "2cacf39dea528bdf84c572af1de772f7"
server_key_iv: "0efb03366190fd2a2eec1d6589379e16"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.540838 2932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.563856 2932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.566151 2932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:53.566262 2932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.566429 2932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:53.566664 2932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:53.566700 2932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.566722 2932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:53.566735 2932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.572266 2932 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:36049
I20260925 19:09:53.572438 3060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:36049 every 8 connection(s)
I20260925 19:09:53.572726 2932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:53.576475 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 2932
I20260925 19:09:53.576639 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2291
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:53.583379 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:37405
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:44679
--webserver_port=41817
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:53.584355 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:41941 (local address 127.27.174.126:44679)
0925 19:09:53.573617 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.573619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.573915 (+ 296us) server_negotiation.cc:407] Connection header received
0925 19:09:53.574433 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.574435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.574448 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.574498 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.575516 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.576146 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578259 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.578483 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578546 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.581326 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.581328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.581328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.581336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.583123 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583477 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.583478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.583478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.583535 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583747 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.583747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.583748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.583862 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.583900 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.584188 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.584267 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 19:09:53.584381 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51413 (local address 127.27.174.124:46095)
0925 19:09:53.573560 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.573562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.573851 (+ 289us) server_negotiation.cc:407] Connection header received
0925 19:09:53.574448 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.574449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.574464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.574494 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.575435 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.576236 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578470 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.578744 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578824 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.581272 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.581273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.581274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.581282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.582770 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583207 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.583208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.583209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.583270 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583550 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.583551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.583551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.583727 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.583776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.584133 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.584235 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:53.584981 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:41941)
0925 19:09:53.573839 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.573851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.574345 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.574905 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.574906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.575092 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.575321 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.575326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.576248 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.576249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.578168 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.578171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.578549 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.578692 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.579180 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.579190 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.581221 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.583201 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.583202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.583396 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.583625 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.583626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.583681 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.583938 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.583939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.583992 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.584497 (+ 505us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.584738 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1377,"thread_start_us":77,"threads_started":1}
I20260925 19:09:53.585177 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.27.174.66:51413)
0925 19:09:53.573718 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.573762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.574327 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.574623 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.574627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.575077 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.575322 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.575327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.576329 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.576330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.578382 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.578385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.578823 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.578824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.578893 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.579333 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.579338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.581172 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.582868 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.582875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.582877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.583111 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.583340 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.583341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.583341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.583471 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.583823 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.583829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.583989 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.584464 (+ 475us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.584671 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":6,"thread_start_us":82,"threads_started":1}
I20260925 19:09:53.585520 3061 heartbeater.cc:347] Connected to a master server at 127.27.174.124:46095
I20260925 19:09:53.585551 3062 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:53.585613 3061 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.586715 3062 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.586908 3061 heartbeater.cc:511] Master 127.27.174.124:46095 requested a full tablet report, sending...
I20260925 19:09:53.586910 3062 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:53.587311 1961 ts_manager.cc:194] Re-registered known tserver with Master: b6949d1218204e7092d266abbbca6b13 (127.27.174.66:36049)
I20260925 19:09:53.587618 2740 ts_manager.cc:194] Registered new tserver with Master: b6949d1218204e7092d266abbbca6b13 (127.27.174.66:36049)
I20260925 19:09:53.587797 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:41941
I20260925 19:09:53.590817 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.27.174.66:51581)
0925 19:09:53.573809 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.573818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.574327 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.574623 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.574627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.575071 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.575322 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.575327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.583164 (+ 7837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.583165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.584036 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.584038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.584617 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.584618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.584684 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.585499 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.585505 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.587766 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.589818 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.589819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.589819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.589996 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.590211 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.590211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.590212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.590264 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.590546 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.590547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.590612 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.590668 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.590747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":33,"thread_start_us":70,"threads_started":1}
I20260925 19:09:53.590932 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.573594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51581 (local address 127.27.174.125:36157)
0925 19:09:53.573642 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.573643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.573861 (+ 218us) server_negotiation.cc:407] Connection header received
0925 19:09:53.574448 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.574449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.574462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.574494 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.575440 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.576228 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.584135 (+ 7907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.584444 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.584494 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.587998 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.587999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.588000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.588008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.589717 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.590076 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.590078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.590078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.590142 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.590328 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.590329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.590330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.590478 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.590553 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.590718 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.590803 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:09:53.591361 3063 heartbeater.cc:347] Connected to a master server at 127.27.174.125:36157
I20260925 19:09:53.591432 3063 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.591507 3063 heartbeater.cc:511] Master 127.27.174.125:36157 requested a full tablet report, sending...
I20260925 19:09:53.591821 2559 ts_manager.cc:194] Registered new tserver with Master: b6949d1218204e7092d266abbbca6b13 (127.27.174.66:36049)
W20260925 19:09:53.666993 3068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:53.667187 3068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:53.667210 3068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:53.668412 3068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:53.668452 3068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:53.668478 3068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:53.669942 3068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:37405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=41817
--tserver_master_addrs=127.27.174.126:44679,127.27.174.125:36157,127.27.174.124:46095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:53.670125 3068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:53.670292 3068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:53.675153 3074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.675169 3073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:53.675169 3076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:53.675338 3068 server_base.cc:1034] running on GCE node
I20260925 19:09:53.675676 3068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:53.675944 3068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:53.677101 3068 hybrid_clock.cc:648] HybridClock initialized: now 1790363393677096 us; error 36 us; skew 500 ppm
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:53.680153 3068 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20260925 19:09:53.680614 3068 webserver.cc:460] Webserver started at http://127.27.174.67:41817/ using document root <none> and password file <none>
I20260925 19:09:53.680850 3068 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:53.680931 3068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:53.682221 3068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:53.682802 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.682982 3068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:53.683081 3068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
format_stamp: "Formatted at 2026-09-25 19:09:52 on dist-test-slave-fcdn"
server_key: "24c700739ff498189f6221f2f2683c0f"
server_key_iv: "d915493783cda394cbeb439de1eb9b2a"
server_key_version: "encryptionkey@0"
I20260925 19:09:53.683408 3068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:53.715097 3068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:53.717218 3068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:53.717372 3068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:53.717621 3068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:09:53.717929 3068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:53.717983 3068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.718026 3068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:53.718060 3068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:53.723881 3068 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37405
I20260925 19:09:53.723939 3196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37405 every 8 connection(s)
I20260925 19:09:53.724313 3068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:53.729579 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 3068
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.124@KRBTEST.COM
Sep 25 19:09:53 dist-test-slave-fcdn krb5kdc[1916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.125@KRBTEST.COM
I20260925 19:09:53.739014 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56785 (local address 127.27.174.124:46095)
0925 19:09:53.725183 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.725185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.725470 (+ 285us) server_negotiation.cc:407] Connection header received
0925 19:09:53.726695 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.726696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.726704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.726777 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.727463 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.728012 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.729548 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.729898 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.729970 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.733678 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.733679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.733680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.733688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.735712 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.737916 (+ 2204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.737917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.737918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.737974 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738277 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.738278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.738278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.738405 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.738480 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.738837 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.738916 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 19:09:53.739014 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52613 (local address 127.27.174.125:36157)
0925 19:09:53.725221 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.725223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.725567 (+ 344us) server_negotiation.cc:407] Connection header received
0925 19:09:53.726484 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.726486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.726500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.726569 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.727460 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.728246 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.730029 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.730359 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.730424 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.735954 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.735956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.735956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.735963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.737697 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738103 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.738104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.738104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.738152 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738348 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.738348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.738349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.738487 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.738527 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.738837 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.738918 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 19:09:53.739431 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:36403 (local address 127.27.174.126:44679)
0925 19:09:53.725305 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.725307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.725383 (+ 76us) server_negotiation.cc:407] Connection header received
0925 19:09:53.726570 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.726571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.726583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.726639 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.727460 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.728243 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.730392 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.730703 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.730755 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.735650 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.735651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.735652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.735660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.737439 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738549 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.738551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.738551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.738614 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738842 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.738843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.738843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.738972 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.739018 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.739250 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.739334 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:09:53.739686 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.27.174.67:56785)
0925 19:09:53.725385 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.725396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.726354 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.726785 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.726786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.727159 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.727349 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.727354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.728112 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.728114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.729455 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.729457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.729984 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.729985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.730126 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.730697 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.730706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.733573 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.735789 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.735796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.735799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.737822 (+ 2023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.738049 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.738050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.738198 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.738492 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.738498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.738663 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.739152 (+ 489us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.739348 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":47,"thread_start_us":99,"threads_started":1}
I20260925 19:09:53.740322 3197 heartbeater.cc:347] Connected to a master server at 127.27.174.124:46095
I20260925 19:09:53.740429 3197 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.740610 3197 heartbeater.cc:511] Master 127.27.174.124:46095 requested a full tablet report, sending...
I20260925 19:09:53.741051 2740 ts_manager.cc:194] Registered new tserver with Master: dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:53.741189 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.27.174.67:52613)
0925 19:09:53.725509 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.725516 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.726354 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.726670 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.726674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.727186 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.727349 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.727355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.728319 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.728320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.729455 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.729457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.731050 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.731050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.731123 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.733716 (+ 2593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.733720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.735525 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.737797 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.737798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.737799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.738021 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.738222 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.738223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.738279 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.738585 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.738597 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.738663 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.740892 (+ 2229us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.740997 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":74,"thread_start_us":72,"threads_started":1}
I20260925 19:09:53.741339 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.730694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.27.174.124:46095)
0925 19:09:53.730977 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.730979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.731167 (+ 188us) server_negotiation.cc:407] Connection header received
0925 19:09:53.731191 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.731192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.731204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.731260 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.732157 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.733001 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.734023 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.734349 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.734410 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.735931 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.735932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.735932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.735938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.739783 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.740218 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.740220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.740220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.740275 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.740467 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.740468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.740469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.740593 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.740689 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.741114 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.741200 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":76,"threads_started":1}
I20260925 19:09:53.741569 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.725024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:36403)
0925 19:09:53.725266 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.725307 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.726344 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.726670 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.726674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.727182 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:53.727349 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.727355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.729556 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.729557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.730292 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.730294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.730836 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.730836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.730899 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.731301 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:53.731305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:53.735229 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:53.737518 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.737518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.737519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.737731 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.738691 (+ 960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:53.738692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:53.738693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:53.738764 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:53.739081 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:53.739082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:53.739130 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.739513 (+ 383us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.739631 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":8,"thread_start_us":79,"threads_started":1}
I20260925 19:09:53.741981 3199 heartbeater.cc:347] Connected to a master server at 127.27.174.125:36157
I20260925 19:09:53.742033 3199 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.742098 3198 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44679
I20260925 19:09:53.742173 3198 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:53.742242 3198 heartbeater.cc:511] Master 127.27.174.126:44679 requested a full tablet report, sending...
I20260925 19:09:53.742118 3199 heartbeater.cc:511] Master 127.27.174.125:36157 requested a full tablet report, sending...
I20260925 19:09:53.742499 1961 ts_manager.cc:194] Re-registered known tserver with Master: dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:53.742539 2559 ts_manager.cc:194] Registered new tserver with Master: dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:53.743011 1961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:36403
I20260925 19:09:53.743799 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.731100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.27.174.125:36157)
0925 19:09:53.731452 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.731454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.731668 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:09:53.732031 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.732032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.732043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.732097 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.732714 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.733301 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.735926 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.736232 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.736291 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.738860 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.738861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.738861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.738868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.740744 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.742478 (+ 1734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.742478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.742479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.742539 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.742754 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.742755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.742755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.742911 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.743004 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.743552 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.743681 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":52,"threads_started":1}
I20260925 19:09:53.745137 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.730725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.27.174.126:44679)
0925 19:09:53.731514 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.731516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.731820 (+ 304us) server_negotiation.cc:407] Connection header received
0925 19:09:53.731846 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.731846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.731861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.731921 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.734024 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.734747 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.737531 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.737877 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.737939 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.741415 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.741416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.741416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.741423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.743995 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.744389 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.744390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.744391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.744451 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.744649 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.744650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.744650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.744757 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.744852 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.744957 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.745033 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":77,"threads_started":1}
I20260925 19:09:53.746662 1961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:53.748241 1961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:53.755853 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37405 (local address 127.0.0.1:42596)
0925 19:09:53.751780 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.751791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.751830 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.752364 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.752365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.752366 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.752408 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.752411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.753583 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.753583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.754686 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.754688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.755580 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.755581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.755682 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.755686 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.755750 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.755789 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:09:53.756036 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.27.174.67:37405)
0925 19:09:53.752188 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.752192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.752201 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:53.752238 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.752240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.752277 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.752358 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.752464 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.753502 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.754779 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.755491 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.755567 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.755668 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.755764 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.755802 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260925 19:09:53.756239 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:36049 (local address 127.0.0.1:33708)
0925 19:09:53.752560 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.752574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.752614 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.752777 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.752778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.752779 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.752806 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.752807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.753915 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.753916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.754712 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.754714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.756019 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.756019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.756084 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.756088 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.756136 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.756175 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":77,"threads_started":1}
I20260925 19:09:53.756500 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.27.174.66:36049)
0925 19:09:53.752083 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.752089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.752638 (+ 549us) server_negotiation.cc:407] Connection header received
0925 19:09:53.752681 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.752684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.752699 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.752783 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.752837 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.753833 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.754829 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.755491 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.755567 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.755666 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.756385 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.756429 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260925 19:09:53.756719 3131 tablet_service.cc:1467] Processing CreateTablet for tablet ec553c45389c42f387201a62218534c9 (DEFAULT_TABLE table=test-table [id=694a2265aefe474786af1ff0e7d4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:53.757081 3131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec553c45389c42f387201a62218534c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:53.757133 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38289 (local address 127.0.0.1:57514)
0925 19:09:53.752162 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.752191 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.752249 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.753603 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.753604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.753606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.753661 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.753663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.754855 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.754856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.755653 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.755655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.756817 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.756818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.756888 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.756892 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.757011 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.757052 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":75,"threads_started":1}
I20260925 19:09:53.757117 2995 tablet_service.cc:1467] Processing CreateTablet for tablet ec553c45389c42f387201a62218534c9 (DEFAULT_TABLE table=test-table [id=694a2265aefe474786af1ff0e7d4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:53.757326 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.751947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.27.174.65:38289)
0925 19:09:53.752111 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.752115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.752304 (+ 189us) server_negotiation.cc:407] Connection header received
0925 19:09:53.752372 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.752374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.752389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.752469 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.753737 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.754772 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.755759 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.756745 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.756818 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.756890 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.757210 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.757255 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260925 19:09:53.757457 2995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec553c45389c42f387201a62218534c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:53.759629 3223 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Bootstrap starting.
I20260925 19:09:53.760278 3223 tablet_bootstrap.cc:654] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:53.760257 3224 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Bootstrap starting.
I20260925 19:09:53.760281 2858 tablet_service.cc:1467] Processing CreateTablet for tablet ec553c45389c42f387201a62218534c9 (DEFAULT_TABLE table=test-table [id=694a2265aefe474786af1ff0e7d4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:53.760587 3223 log.cc:824] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:53.760610 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec553c45389c42f387201a62218534c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:53.760895 3224 tablet_bootstrap.cc:654] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:53.761214 3224 log.cc:824] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:53.761322 3223 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: No bootstrap required, opened a new log
I20260925 19:09:53.761394 3223 ts_tablet_manager.cc:1397] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:53.761956 3224 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: No bootstrap required, opened a new log
I20260925 19:09:53.762018 3224 ts_tablet_manager.cc:1397] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:53.762732 3223 raft_consensus.cc:348] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.762843 3223 raft_consensus.cc:374] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:53.762863 3223 raft_consensus.cc:729] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3b1d8f08084d9aaac8bfaaa429399f, State: Initialized, Role: FOLLOWER
I20260925 19:09:53.762979 3223 consensus_queue.cc:260] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.763221 3223 ts_tablet_manager.cc:1428] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:53.763471 3198 heartbeater.cc:503] Master 127.27.174.126:44679 was elected leader, sending a full tablet report...
I20260925 19:09:53.764519 3224 raft_consensus.cc:348] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.764663 3224 raft_consensus.cc:374] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:53.764699 3224 raft_consensus.cc:729] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6949d1218204e7092d266abbbca6b13, State: Initialized, Role: FOLLOWER
I20260925 19:09:53.764715 3228 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Bootstrap starting.
I20260925 19:09:53.764812 3224 consensus_queue.cc:260] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.765054 3224 ts_tablet_manager.cc:1428] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:53.765103 3062 heartbeater.cc:503] Master 127.27.174.126:44679 was elected leader, sending a full tablet report...
I20260925 19:09:53.765677 3228 tablet_bootstrap.cc:654] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:53.766010 3228 log.cc:824] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:53.766667 3228 tablet_bootstrap.cc:492] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: No bootstrap required, opened a new log
I20260925 19:09:53.766719 3228 ts_tablet_manager.cc:1397] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:53.768628 3228 raft_consensus.cc:348] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.768749 3228 raft_consensus.cc:374] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:53.768770 3228 raft_consensus.cc:729] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832df0ab2375467da11a100246364c39, State: Initialized, Role: FOLLOWER
I20260925 19:09:53.768889 3228 consensus_queue.cc:260] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.769160 3228 ts_tablet_manager.cc:1428] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:09:53.769188 2925 heartbeater.cc:503] Master 127.27.174.126:44679 was elected leader, sending a full tablet report...
W20260925 19:09:53.823509 3064 tablet.cc:2369] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:53.922941 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.919914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:40096)
0925 19:09:53.919979 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.919992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.920058 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.920342 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.920343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.920345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.920395 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.920397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.921409 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.921411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.922159 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.922161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.922751 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.922751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.922797 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.922800 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.922844 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.922883 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:09:53.922936 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.920056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.27.174.124:46095)
0925 19:09:53.920143 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.920145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.920154 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:53.920192 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.920192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.920213 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.920291 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.920528 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.921305 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.922257 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.922675 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.922739 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.922796 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.922849 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.922878 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 19:09:53.923375 2755 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 19:09:53.924384 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5633ebab9347208c5c00659f62dd50. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.924522 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: LEARNER
W20260925 19:09:53.929693 2927 tablet.cc:2369] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:53.951769 3229 raft_consensus.cc:482] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:53.951892 3229 raft_consensus.cc:504] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.952291 3229 leader_election.cc:288] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405), 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
I20260925 19:09:53.956270 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.952587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44357 (local address 127.27.174.65:38289)
0925 19:09:53.952643 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.952644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.952692 (+ 48us) server_negotiation.cc:407] Connection header received
0925 19:09:53.952721 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.952722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.952731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.952787 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.953018 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.953765 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955334 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.955900 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955984 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.955992 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.956161 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.956200 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:53.956270 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.952587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45467 (local address 127.27.174.67:37405)
0925 19:09:53.952633 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.952635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.952648 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:09:53.952721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.952722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.952731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.952787 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:53.953022 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.953778 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955334 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.955917 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955984 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.955992 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.956165 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.956200 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:53.956297 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.952517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37405 (local address 127.27.174.66:45467)
0925 19:09:53.952604 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.952613 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.952644 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.952807 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.952807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.952809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.952924 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.952926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.953879 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.953881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.955219 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.955223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.955990 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.956076 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.956078 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.956135 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.956187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":142}
I20260925 19:09:53.956310 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.952511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38289 (local address 127.27.174.66:44357)
0925 19:09:53.952560 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:53.952573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:53.952632 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:53.952807 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:53.952807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:53.952809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:53.952913 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.952941 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.953879 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.953881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.955219 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:53.955223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.955990 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.955991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:53.956069 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.956073 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:53.956135 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:09:53.956187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":83}
I20260925 19:09:53.956702 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec553c45389c42f387201a62218534c9" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832df0ab2375467da11a100246364c39" is_pre_election: true
I20260925 19:09:53.956880 2878 raft_consensus.cc:2454] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 0.
I20260925 19:09:53.956875 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec553c45389c42f387201a62218534c9" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" is_pre_election: true
I20260925 19:09:53.956991 3151 raft_consensus.cc:2454] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 0.
I20260925 19:09:53.957111 2949 leader_election.cc:302] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 832df0ab2375467da11a100246364c39, b6949d1218204e7092d266abbbca6b13; no voters:
I20260925 19:09:53.957264 3229 raft_consensus.cc:2781] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:53.957348 3229 raft_consensus.cc:482] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:53.957383 3229 raft_consensus.cc:3037] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:53.958119 3229 raft_consensus.cc:504] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.958274 3229 leader_election.cc:288] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 election: Requested vote from peers dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405), 832df0ab2375467da11a100246364c39 (127.27.174.65:38289)
I20260925 19:09:53.958405 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec553c45389c42f387201a62218534c9" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
I20260925 19:09:53.958456 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec553c45389c42f387201a62218534c9" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832df0ab2375467da11a100246364c39"
I20260925 19:09:53.958527 3151 raft_consensus.cc:3037] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:53.958529 2878 raft_consensus.cc:3037] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:53.959200 2878 raft_consensus.cc:2454] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 1.
I20260925 19:09:53.959223 3151 raft_consensus.cc:2454] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 1.
I20260925 19:09:53.959357 2949 leader_election.cc:302] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 832df0ab2375467da11a100246364c39, b6949d1218204e7092d266abbbca6b13; no voters:
I20260925 19:09:53.959482 3229 raft_consensus.cc:2781] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:53.959682 3229 raft_consensus.cc:686] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 1 LEADER]: Becoming Leader. State: Replica: b6949d1218204e7092d266abbbca6b13, State: Running, Role: LEADER
I20260925 19:09:53.959790 3229 consensus_queue.cc:237] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } }
I20260925 19:09:53.960557 1961 catalog_manager.cc:5495] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6949d1218204e7092d266abbbca6b13 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "b6949d1218204e7092d266abbbca6b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:53.961210 3239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: NON_VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: true } } }
I20260925 19:09:53.961305 3239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.961849 2709 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: attempting to promote NON_VOTER 2e1b8769edef4147b426d62574254220 to VOTER
I20260925 19:09:53.961990 2709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:53.962416 2755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 LEARNER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 19:09:53.962448 2569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Refusing update from remote peer 2c5633ebab9347208c5c00659f62dd50: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 19:09:53.962627 3239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 19:09:53.962718 3241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 19:09:53.963346 2520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2e1b8769edef4147b426d62574254220 (127.27.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.963457 2755 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2e1b8769edef4147b426d62574254220 (127.27.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.964304 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.964401 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.964367 3241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.964459 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.964464 3241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.964526 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:53.964507 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.964597 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:53.964746 2569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2e1b8769edef4147b426d62574254220 (127.27.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.965440 3242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c5633ebab9347208c5c00659f62dd50" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:53.965570 3242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:09:53.975065 3200 tablet.cc:2369] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:53.981163 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.978869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.27.174.66:36049)
0925 19:09:53.978907 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.978908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.978916 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:53.978968 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.978969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.978985 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.979044 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:53.979138 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.979785 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.980472 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.980720 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.980798 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.980915 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:53.981008 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:53.981065 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:53.981093 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 19:09:53.981109 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:53.982929 3151 raft_consensus.cc:1261] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Refusing update from remote peer b6949d1218204e7092d266abbbca6b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:53.982930 2878 raft_consensus.cc:1261] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Refusing update from remote peer b6949d1218204e7092d266abbbca6b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:53.983278 3229 consensus_queue.cc:1035] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:53.983440 3237 consensus_queue.cc:1035] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:53.986184 3244 mvcc.cc:205] Tried to move back new op lower bound from 7333328461751177216 to 7333328461659807744. Current Snapshot: MvccSnapshot[applied={T|T < 7333328461751177216}]
I20260925 19:09:53.986925 3245 mvcc.cc:205] Tried to move back new op lower bound from 7333328461751177216 to 7333328461659807744. Current Snapshot: MvccSnapshot[applied={T|T < 7333328461751177216}]
I20260925 19:09:53.988438 1960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:53.988538 1960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:53.988866 3246 mvcc.cc:205] Tried to move back new op lower bound from 7333328461751177216 to 7333328461659807744. Current Snapshot: MvccSnapshot[applied={T|T < 7333328461751177216}]
I20260925 19:09:53.989995 1960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5633ebab9347208c5c00659f62dd50: Sending DeleteTablet for 3 replicas of tablet ec553c45389c42f387201a62218534c9
I20260925 19:09:53.990338 2858 tablet_service.cc:1514] Processing DeleteTablet for tablet ec553c45389c42f387201a62218534c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:53 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57514
I20260925 19:09:53.990525 3253 tablet_replica.cc:330] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: stopping tablet replica
I20260925 19:09:53.990549 3131 tablet_service.cc:1514] Processing DeleteTablet for tablet ec553c45389c42f387201a62218534c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:53 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:42596
I20260925 19:09:53.990633 3253 raft_consensus.cc:2229] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:53.990751 3253 raft_consensus.cc:2258] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:53.990828 2995 tablet_service.cc:1514] Processing DeleteTablet for tablet ec553c45389c42f387201a62218534c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:53 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33708
I20260925 19:09:53.991194 3253 ts_tablet_manager.cc:1905] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:53.992363 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 1929
I20260925 19:09:53.993281 3253 ts_tablet_manager.cc:1918] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:53.993350 3253 log.cc:1196] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/ec553c45389c42f387201a62218534c9
I20260925 19:09:53.993605 3253 ts_tablet_manager.cc:1939] T ec553c45389c42f387201a62218534c9 P 832df0ab2375467da11a100246364c39: Deleting consensus metadata
W20260925 19:09:53.998157 2811 connection.cc:535] server connection from 127.0.0.1:57514 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:09:53.998227 3254 tablet_replica.cc:330] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: stopping tablet replica
I20260925 19:09:53.998319 3254 raft_consensus.cc:2229] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:53.998418 3254 raft_consensus.cc:2258] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:53.998571 3255 tablet_replica.cc:330] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: stopping tablet replica
I20260925 19:09:53.998692 3255 raft_consensus.cc:2229] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:53.998788 3255 raft_consensus.cc:2258] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:53.999239 3254 ts_tablet_manager.cc:1905] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:54.000259 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:53.991440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.27.174.124:46095)
0925 19:09:53.991478 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:53.991479 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:53.991487 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:53.991507 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:53.991507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:53.991516 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:53.991578 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:53.992346 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.993063 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.994734 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:53.995026 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:53.995068 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:53.996790 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:53.996791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:53.996791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:53.996798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:53.998472 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.999390 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.999391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.999392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.999447 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:53.999679 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:53.999679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:53.999680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:53.999808 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:53.999883 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:54.000092 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 19:09:54.000169 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:54.001173 3255 ts_tablet_manager.cc:1905] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:54.002839 3255 ts_tablet_manager.cc:1918] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:54.002907 3255 log.cc:1196] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/ec553c45389c42f387201a62218534c9
I20260925 19:09:54.003172 3255 ts_tablet_manager.cc:1939] T ec553c45389c42f387201a62218534c9 P b6949d1218204e7092d266abbbca6b13: Deleting consensus metadata
W20260925 19:09:54.003496 2949 connection.cc:441] server connection from 127.0.0.1:33708 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33708 (request call id 1) could send its response
I20260925 19:09:54.003542 3254 ts_tablet_manager.cc:1918] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:54.003706 3254 log.cc:1196] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/ec553c45389c42f387201a62218534c9
I20260925 19:09:54.003960 3254 ts_tablet_manager.cc:1939] T ec553c45389c42f387201a62218534c9 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting consensus metadata
W20260925 19:09:54.004289 3084 connection.cc:441] server connection from 127.0.0.1:42596 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42596 (request call id 1) could send its response
I20260925 19:09:54.008809 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:54.000194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.27.174.125:36157)
0925 19:09:54.000235 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:54.000237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:54.000903 (+ 666us) server_negotiation.cc:407] Connection header received
0925 19:09:54.001028 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:54.001029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:54.001043 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:54.001087 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:54.002198 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:54.002936 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:54.004332 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:54.004595 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:54.004631 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:54.005800 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:54.005802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:54.005802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:54.005809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:54.007484 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:54.007951 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:54.007953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:54.007953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:54.008011 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:54.008232 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:54.008233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:54.008234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:54.008376 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:54.008478 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:54.008628 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:09:54.008714 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:54.011282 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:54.000662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.27.174.124:46095)
0925 19:09:54.000694 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:54.000696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:54.004247 (+ 3551us) server_negotiation.cc:407] Connection header received
0925 19:09:54.004294 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:54.004295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:54.004304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:54.004359 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:54.005372 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:54.006101 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:54.006933 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:54.007139 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:54.007214 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:54.008549 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:54.008550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:54.008550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:54.008557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:54.010109 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:54.010485 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:54.010486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:54.010486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:54.010560 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:54.010826 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:54.010827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:54.010827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:54.010960 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:54.011049 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:54.011171 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:09:54.011225 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:09:54.987937 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:54.987654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:43207)
0925 19:09:54.987796 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:54.987881 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
W20260925 19:09:54.988202 3198 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:54.989223 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:54.988943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:34397)
0925 19:09:54.989110 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:54.989174 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":50,"threads_started":1}
I20260925 19:09:54.989283 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:54.989049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:50341)
0925 19:09:54.989172 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:54.989239 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260925 19:09:54.989511 3062 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
W20260925 19:09:54.989521 2925 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.490079 3267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c5633ebab9347208c5c00659f62dd50)
I20260925 19:09:55.490146 3268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c5633ebab9347208c5c00659f62dd50)
I20260925 19:09:55.490262 3267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:55.490274 3268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:55.490552 3267 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125:36157)
I20260925 19:09:55.490871 3268 leader_election.cc:288] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095)
I20260925 19:09:55.490890 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52618)
0925 19:09:55.490783 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.490848 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260925 19:09:55.491009 2730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.491127 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52624)
0925 19:09:55.491010 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.491085 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":49,"threads_started":1}
W20260925 19:09:55.491232 2544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.491484 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.491318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52630)
0925 19:09:55.491425 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.491455 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 19:09:55.491636 2730 leader_election.cc:334] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.491860 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.491775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52642)
0925 19:09:55.491805 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.491836 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 19:09:55.491989 2544 leader_election.cc:334] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.494012 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:46095 (local address 127.0.0.1:40120)
0925 19:09:55.491070 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.491079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:55.491134 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:55.491506 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:55.491507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:55.491509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:55.491555 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:55.491558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.492458 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.492459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:55.493197 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:55.493198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.493757 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:55.493823 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:55.493826 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:55.493901 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:09:55.493951 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":36,"threads_started":1}
I20260925 19:09:55.494017 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.27.174.124:46095)
0925 19:09:55.491362 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:55.491364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:55.491371 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:55.491392 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:55.491393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:55.491407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:55.491642 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:55.491676 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.492392 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493285 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.493671 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493756 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:55.493766 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:55.493908 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:09:55.493947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":441,"thread_start_us":39,"threads_started":1}
I20260925 19:09:55.494129 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:36157 (local address 127.0.0.1:58436)
0925 19:09:55.490895 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.490907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:55.491104 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:55.491324 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:55.491325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:55.491328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:55.491392 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:55.491394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.492562 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.492563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:55.493310 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:55.493312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.493875 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:55.493941 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:55.493943 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:55.494017 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:09:55.494060 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":40,"threads_started":1}
I20260925 19:09:55.494123 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.490806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.27.174.125:36157)
0925 19:09:55.490977 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:55.490979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:55.490991 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:55.491217 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:55.491218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:55.491226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:55.491281 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:55.491487 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.492483 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493418 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:55.493794 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:55.493877 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:55.493884 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:55.494024 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 19:09:55.494060 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":46,"threads_started":1}
I20260925 19:09:55.494618 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e1b8769edef4147b426d62574254220" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "281ff51e1aa541ed973c67f62b3415ab" is_pre_election: true
I20260925 19:09:55.494648 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "281ff51e1aa541ed973c67f62b3415ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e1b8769edef4147b426d62574254220" is_pre_election: true
I20260925 19:09:55.494733 2569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1b8769edef4147b426d62574254220 in term 1.
I20260925 19:09:55.494728 2755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 281ff51e1aa541ed973c67f62b3415ab in term 1.
I20260925 19:09:55.494875 2731 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 281ff51e1aa541ed973c67f62b3415ab, 2e1b8769edef4147b426d62574254220; no voters: 2c5633ebab9347208c5c00659f62dd50
I20260925 19:09:55.494939 2543 leader_election.cc:302] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 281ff51e1aa541ed973c67f62b3415ab, 2e1b8769edef4147b426d62574254220; no voters: 2c5633ebab9347208c5c00659f62dd50
I20260925 19:09:55.494988 3267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:09:55.495040 3267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c5633ebab9347208c5c00659f62dd50)
I20260925 19:09:55.495048 3268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:09:55.495069 3267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:09:55.495079 3268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c5633ebab9347208c5c00659f62dd50)
I20260925 19:09:55.495098 3268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:09:55.495802 3267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:55.495814 3268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:55.495935 3267 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 election: Requested vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 281ff51e1aa541ed973c67f62b3415ab (127.27.174.125:36157)
I20260925 19:09:55.495946 3268 leader_election.cc:288] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 election: Requested vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095)
I20260925 19:09:55.496114 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.496003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52658)
0925 19:09:55.496056 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.496086 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:09:55.496096 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e1b8769edef4147b426d62574254220" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "281ff51e1aa541ed973c67f62b3415ab"
I20260925 19:09:55.496197 2569 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e1b8769edef4147b426d62574254220 in current term 2: Already voted for candidate 281ff51e1aa541ed973c67f62b3415ab in this term.
I20260925 19:09:55.496243 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.496084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52674)
0925 19:09:55.496172 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.496192 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 19:09:55.496340 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "281ff51e1aa541ed973c67f62b3415ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e1b8769edef4147b426d62574254220"
I20260925 19:09:55.496377 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.496298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52678)
0925 19:09:55.496330 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.496355 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 19:09:55.496433 2755 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 281ff51e1aa541ed973c67f62b3415ab in current term 2: Already voted for candidate 2e1b8769edef4147b426d62574254220 in this term.
W20260925 19:09:55.496488 2730 leader_election.cc:334] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.496490 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.496418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52680)
0925 19:09:55.496451 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.496473 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 19:09:55.496519 2730 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e1b8769edef4147b426d62574254220; no voters: 281ff51e1aa541ed973c67f62b3415ab, 2c5633ebab9347208c5c00659f62dd50
W20260925 19:09:55.496572 2544 leader_election.cc:334] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:55.496614 3267 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 19:09:55.496685 2543 leader_election.cc:302] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 281ff51e1aa541ed973c67f62b3415ab; no voters: 2c5633ebab9347208c5c00659f62dd50, 2e1b8769edef4147b426d62574254220
I20260925 19:09:55.496773 3268 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 19:09:55.988740 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.988540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:38993)
0925 19:09:55.988661 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.988699 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260925 19:09:55.989971 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.989783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:46989)
0925 19:09:55.989893 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.989928 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260925 19:09:55.990041 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:55.989779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:60217)
0925 19:09:55.989951 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:55.989997 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260925 19:09:56.989383 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:56.989154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:42283)
0925 19:09:56.989302 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:56.989341 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260925 19:09:56.990607 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:56.990398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:33167)
0925 19:09:56.990529 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:56.990573 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260925 19:09:56.990635 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:56.990445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:59615)
0925 19:09:56.990544 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:56.990574 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260925 19:09:57.198593 3283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:57.198742 3283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:57.198961 3283 leader_election.cc:288] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095)
I20260925 19:09:57.199201 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "281ff51e1aa541ed973c67f62b3415ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e1b8769edef4147b426d62574254220" is_pre_election: true
I20260925 19:09:57.199321 2755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 281ff51e1aa541ed973c67f62b3415ab in term 2.
I20260925 19:09:57.199503 2543 leader_election.cc:302] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 281ff51e1aa541ed973c67f62b3415ab, 2e1b8769edef4147b426d62574254220; no voters:
I20260925 19:09:57.199635 3283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 19:09:57.199679 3283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:57.199698 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.199075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52948)
0925 19:09:57.199581 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.199650 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":58,"threads_started":1}
I20260925 19:09:57.199712 3283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:09:57.200161 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.200051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52964)
0925 19:09:57.200114 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.200139 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 19:09:57.200371 2544 leader_election.cc:334] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:57.200907 3283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:57.201225 3283 leader_election.cc:288] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 election: Requested vote from peers 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679), 2e1b8769edef4147b426d62574254220 (127.27.174.124:46095)
I20260925 19:09:57.201339 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.201165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52974)
0925 19:09:57.201212 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.201237 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:09:57.201334 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "281ff51e1aa541ed973c67f62b3415ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e1b8769edef4147b426d62574254220"
I20260925 19:09:57.201396 2755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:09:57.201980 2755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 281ff51e1aa541ed973c67f62b3415ab in term 3.
I20260925 19:09:57.202167 2543 leader_election.cc:302] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 281ff51e1aa541ed973c67f62b3415ab, 2e1b8769edef4147b426d62574254220; no voters:
I20260925 19:09:57.202265 3283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:09:57.202370 3283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [term 3 LEADER]: Becoming Leader. State: Replica: 281ff51e1aa541ed973c67f62b3415ab, State: Running, Role: LEADER
I20260925 19:09:57.202410 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.202330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52980)
0925 19:09:57.202363 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.202383 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 19:09:57.202464 2544 leader_election.cc:334] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
I20260925 19:09:57.202451 3283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } }
I20260925 19:09:57.202816 3286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 281ff51e1aa541ed973c67f62b3415ab. Latest consensus state: current_term: 3 leader_uuid: "281ff51e1aa541ed973c67f62b3415ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:57.202915 3286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:57.203107 3288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:57.203660 3288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:57.203758 3288 catalog_manager.cc:1259] Loaded cluster ID: 1104f78ee60c4efaa35192681f7ad6b4
I20260925 19:09:57.203796 3288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:57.204031 3288 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:57.204167 3288 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Loaded TSK: 0
I20260925 19:09:57.204352 3288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:57.248775 2558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:57.250294 2558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:57.251333 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.251204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:52994)
0925 19:09:57.251269 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.251309 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:09:57.251356 2755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [term 3 FOLLOWER]: Refusing update from remote peer 281ff51e1aa541ed973c67f62b3415ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 19:09:57.251691 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.251577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.0.0.1:53008)
0925 19:09:57.251623 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.251674 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 19:09:57.251650 3286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 19:09:57.251804 2544 consensus_peers.cc:487] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab -> Peer 2c5633ebab9347208c5c00659f62dd50 (127.27.174.126:44679): Couldn't send request to peer 2c5633ebab9347208c5c00659f62dd50. Status: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 19:09:57.252332 3290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 281ff51e1aa541ed973c67f62b3415ab. Latest consensus state: current_term: 3 leader_uuid: "281ff51e1aa541ed973c67f62b3415ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:57.252441 3290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:57.252894 3283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "281ff51e1aa541ed973c67f62b3415ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:57.252991 3283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:57.253301 3290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "281ff51e1aa541ed973c67f62b3415ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2c5633ebab9347208c5c00659f62dd50" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44679 } } peers { permanent_uuid: "281ff51e1aa541ed973c67f62b3415ab" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 36157 } attrs { promote: false } } peers { permanent_uuid: "2e1b8769edef4147b426d62574254220" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 46095 } attrs { promote: false } } }
I20260925 19:09:57.253389 3290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1b8769edef4147b426d62574254220 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:57.253434 3291 mvcc.cc:205] Tried to move back new op lower bound from 7333328475139358720 to 7333328474941919232. Current Snapshot: MvccSnapshot[applied={T|T < 7333328475139358720 or (T in {7333328475139358720})}]
I20260925 19:09:57.257858 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38289 (local address 127.0.0.1:57520)
0925 19:09:57.254912 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.254922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:57.254972 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:57.255354 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:57.255354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:57.255356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:57.255445 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.255447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.256157 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.256158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.256868 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.256869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257623 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.257685 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.257689 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:57.257746 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:09:57.257786 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
I20260925 19:09:57.257867 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.27.174.65:38289)
0925 19:09:57.255098 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:57.255100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:57.255108 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:57.255133 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:57.255134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:57.255145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:57.255277 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:57.255519 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.256074 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.256965 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257534 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257614 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.257626 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:57.257760 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 19:09:57.257797 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":56,"threads_started":1}
I20260925 19:09:57.258169 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:36049 (local address 127.0.0.1:33716)
0925 19:09:57.254775 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.254785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:57.254838 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:57.255026 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:57.255027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:57.255029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:57.255080 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.255083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.256038 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.256039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.257219 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.257222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257943 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.258013 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.258016 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:57.258065 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:09:57.258104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":2}
I20260925 19:09:57.258193 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.27.174.66:36049)
0925 19:09:57.254833 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:57.254835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:57.254845 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:57.254917 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:57.254918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:57.254931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:57.254995 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:57.255240 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.255972 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257304 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257839 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257899 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.257910 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:57.258072 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 19:09:57.258111 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
I20260925 19:09:57.258396 2995 tablet_service.cc:1467] Processing CreateTablet for tablet 2600d3f008784be29531b6ee73ee6e02 (DEFAULT_TABLE table=test-table [id=ba64b09a024c4cf08927e902ceb45c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:57.258445 2858 tablet_service.cc:1467] Processing CreateTablet for tablet 2600d3f008784be29531b6ee73ee6e02 (DEFAULT_TABLE table=test-table [id=ba64b09a024c4cf08927e902ceb45c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:57.258554 2995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2600d3f008784be29531b6ee73ee6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:57.258575 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2600d3f008784be29531b6ee73ee6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:57.259655 3300 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: Bootstrap starting.
I20260925 19:09:57.259719 3301 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Bootstrap starting.
I20260925 19:09:57.259991 3300 tablet_bootstrap.cc:654] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:57.260046 3301 tablet_bootstrap.cc:654] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:57.260608 3300 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: No bootstrap required, opened a new log
I20260925 19:09:57.260608 3301 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: No bootstrap required, opened a new log
I20260925 19:09:57.260661 3300 ts_tablet_manager.cc:1397] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:57.260661 3301 ts_tablet_manager.cc:1397] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:57.260730 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37405 (local address 127.0.0.1:42612)
0925 19:09:57.255269 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.255278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:57.255318 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:57.256209 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:57.256210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:57.256212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:57.256256 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.256258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257099 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.257100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.258293 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:57.258296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.260502 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.260503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:57.260567 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.260571 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:57.260595 (+ 24us) client_negotiation.cc:240] Negotiation successful
0925 19:09:57.260624 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260925 19:09:57.260778 3301 raft_consensus.cc:348] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.260795 3300 raft_consensus.cc:348] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.260839 3301 raft_consensus.cc:374] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:57.260864 3301 raft_consensus.cc:729] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6949d1218204e7092d266abbbca6b13, State: Initialized, Role: FOLLOWER
I20260925 19:09:57.260862 3300 raft_consensus.cc:374] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:57.260885 3300 raft_consensus.cc:729] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832df0ab2375467da11a100246364c39, State: Initialized, Role: FOLLOWER
I20260925 19:09:57.260914 3301 consensus_queue.cc:260] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.260969 3300 consensus_queue.cc:260] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.261047 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.254938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.27.174.67:37405)
0925 19:09:57.255176 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:57.255179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:57.256053 (+ 874us) server_negotiation.cc:407] Connection header received
0925 19:09:57.256080 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:57.256081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:57.256091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:57.256168 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:57.256327 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.257021 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.259784 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.260394 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.260898 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.260913 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:57.260932 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:09:57.260974 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":53,"threads_started":1}
I20260925 19:09:57.261163 3301 ts_tablet_manager.cc:1428] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:57.261211 3300 ts_tablet_manager.cc:1428] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:57.261227 3131 tablet_service.cc:1467] Processing CreateTablet for tablet 2600d3f008784be29531b6ee73ee6e02 (DEFAULT_TABLE table=test-table [id=ba64b09a024c4cf08927e902ceb45c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:57.261328 3131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2600d3f008784be29531b6ee73ee6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:57.261663 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.261497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:39893)
0925 19:09:57.261604 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.261635 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 19:09:57.261968 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.261844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:48625)
0925 19:09:57.261907 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.261941 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 19:09:57.262531 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.262349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:45227)
0925 19:09:57.262475 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.262502 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 19:09:57.262658 2926 heartbeater.cc:503] Master 127.27.174.125:36157 was elected leader, sending a full tablet report...
I20260925 19:09:57.262893 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.262812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:39587)
0925 19:09:57.262848 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.262870 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 19:09:57.263012 3306 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Bootstrap starting.
I20260925 19:09:57.263084 3063 heartbeater.cc:503] Master 127.27.174.125:36157 was elected leader, sending a full tablet report...
I20260925 19:09:57.263365 3306 tablet_bootstrap.cc:654] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:57.263943 3306 tablet_bootstrap.cc:492] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: No bootstrap required, opened a new log
I20260925 19:09:57.264004 3306 ts_tablet_manager.cc:1397] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:57.264145 3306 raft_consensus.cc:348] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.264217 3306 raft_consensus.cc:374] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:57.264240 3306 raft_consensus.cc:729] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3b1d8f08084d9aaac8bfaaa429399f, State: Initialized, Role: FOLLOWER
I20260925 19:09:57.264292 3306 consensus_queue.cc:260] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.264467 3306 ts_tablet_manager.cc:1428] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:57.264763 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.264631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:60365)
0925 19:09:57.264688 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.264733 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:09:57.265043 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.264960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:55149)
0925 19:09:57.264992 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.265019 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:09:57.265131 3199 heartbeater.cc:503] Master 127.27.174.125:36157 was elected leader, sending a full tablet report...
I20260925 19:09:57.392429 3304 raft_consensus.cc:482] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:57.392565 3304 raft_consensus.cc:504] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.392771 3304 leader_election.cc:288] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 832df0ab2375467da11a100246364c39 (127.27.174.65:38289), dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:57.393049 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2600d3f008784be29531b6ee73ee6e02" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832df0ab2375467da11a100246364c39" is_pre_election: true
I20260925 19:09:57.393047 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2600d3f008784be29531b6ee73ee6e02" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" is_pre_election: true
I20260925 19:09:57.393151 3151 raft_consensus.cc:2454] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 0.
I20260925 19:09:57.393155 2878 raft_consensus.cc:2454] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 0.
I20260925 19:09:57.393316 2951 leader_election.cc:302] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6949d1218204e7092d266abbbca6b13, dc3b1d8f08084d9aaac8bfaaa429399f; no voters:
I20260925 19:09:57.393428 3304 raft_consensus.cc:2781] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:57.393469 3304 raft_consensus.cc:482] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:57.393488 3304 raft_consensus.cc:3037] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:57.394184 3304 raft_consensus.cc:504] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.394321 3304 leader_election.cc:288] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 election: Requested vote from peers 832df0ab2375467da11a100246364c39 (127.27.174.65:38289), dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405)
I20260925 19:09:57.394490 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2600d3f008784be29531b6ee73ee6e02" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f"
I20260925 19:09:57.394500 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2600d3f008784be29531b6ee73ee6e02" candidate_uuid: "b6949d1218204e7092d266abbbca6b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832df0ab2375467da11a100246364c39"
I20260925 19:09:57.394584 2878 raft_consensus.cc:3037] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:57.394587 3151 raft_consensus.cc:3037] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:57.395190 3151 raft_consensus.cc:2454] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 1.
I20260925 19:09:57.395213 2878 raft_consensus.cc:2454] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6949d1218204e7092d266abbbca6b13 in term 1.
I20260925 19:09:57.395380 2951 leader_election.cc:302] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6949d1218204e7092d266abbbca6b13, dc3b1d8f08084d9aaac8bfaaa429399f; no voters:
I20260925 19:09:57.395489 3304 raft_consensus.cc:2781] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:57.395691 3304 raft_consensus.cc:686] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 1 LEADER]: Becoming Leader. State: Replica: b6949d1218204e7092d266abbbca6b13, State: Running, Role: LEADER
I20260925 19:09:57.395897 3304 consensus_queue.cc:237] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } }
I20260925 19:09:57.396070 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.395927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:37381)
0925 19:09:57.395998 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.396037 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 19:09:57.396636 2559 catalog_manager.cc:5495] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6949d1218204e7092d266abbbca6b13 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "b6949d1218204e7092d266abbbca6b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6949d1218204e7092d266abbbca6b13" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 36049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:57.402410 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.400365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.27.174.66:36049)
0925 19:09:57.400406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:57.400408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:57.400417 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:57.400498 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:57.400499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:57.400507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:57.400555 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:57.400669 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.401125 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.401901 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:57.402093 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:57.402142 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:57.402249 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:57.402271 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:57.402303 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:57.402352 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 19:09:57.402366 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:57.403141 3151 raft_consensus.cc:1261] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Refusing update from remote peer b6949d1218204e7092d266abbbca6b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:57.403146 2878 raft_consensus.cc:1261] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Refusing update from remote peer b6949d1218204e7092d266abbbca6b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:57.403335 3309 consensus_queue.cc:1035] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc3b1d8f08084d9aaac8bfaaa429399f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:57.403461 3304 consensus_queue.cc:1035] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832df0ab2375467da11a100246364c39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:57.404261 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.404153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:39073)
0925 19:09:57.404196 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.404227 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 19:09:57.404503 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.404314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:54231)
0925 19:09:57.404447 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.404475 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 19:09:57.405615 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.405456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:55903)
0925 19:09:57.405547 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.405589 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 19:09:57.406045 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.405889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.65:54627)
0925 19:09:57.405992 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.406017 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 19:09:57.406193 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.405956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.66:55665)
0925 19:09:57.406145 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.406172 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260925 19:09:57.406227 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.406087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44679 (local address 127.27.174.67:40469)
0925 19:09:57.406169 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.406193 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 19:09:57.407814 2559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:57.407905 2559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:57.408933 2559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 281ff51e1aa541ed973c67f62b3415ab: Sending DeleteTablet for 3 replicas of tablet 2600d3f008784be29531b6ee73ee6e02
I20260925 19:09:57.409207 2995 tablet_service.cc:1514] Processing DeleteTablet for tablet 2600d3f008784be29531b6ee73ee6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:57 UTC) from {username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:33716
I20260925 19:09:57.409246 3131 tablet_service.cc:1514] Processing DeleteTablet for tablet 2600d3f008784be29531b6ee73ee6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:57 UTC) from {username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:42612
I20260925 19:09:57.409329 2858 tablet_service.cc:1514] Processing DeleteTablet for tablet 2600d3f008784be29531b6ee73ee6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:57 UTC) from {username='oryx', principal='oryx/127.27.174.125@KRBTEST.COM'} at 127.0.0.1:57520
I20260925 19:09:57.409449 3321 tablet_replica.cc:330] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: stopping tablet replica
I20260925 19:09:57.409497 3323 tablet_replica.cc:330] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39: stopping tablet replica
I20260925 19:09:57.409507 3321 raft_consensus.cc:2229] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:57.409539 3323 raft_consensus.cc:2229] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:57.409574 3321 raft_consensus.cc:2258] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:57.409699 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2706
I20260925 19:09:57.409583 3323 raft_consensus.cc:2258] T 2600d3f008784be29531b6ee73ee6e02 P 832df0ab2375467da11a100246364c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:57.409914 3322 tablet_replica.cc:330] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: stopping tablet replica
I20260925 19:09:57.409961 3322 raft_consensus.cc:2229] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:57.410015 3322 raft_consensus.cc:2258] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:57.410656 3322 ts_tablet_manager.cc:1905] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:57.410694 3321 ts_tablet_manager.cc:1905] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:57.412200 3321 ts_tablet_manager.cc:1918] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:57.412281 3321 log.cc:1196] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/2600d3f008784be29531b6ee73ee6e02
I20260925 19:09:57.412510 3322 ts_tablet_manager.cc:1918] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:57.412576 3321 ts_tablet_manager.cc:1939] T 2600d3f008784be29531b6ee73ee6e02 P b6949d1218204e7092d266abbbca6b13: Deleting consensus metadata
I20260925 19:09:57.412583 3322 log.cc:1196] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/2600d3f008784be29531b6ee73ee6e02
I20260925 19:09:57.412801 3322 ts_tablet_manager.cc:1939] T 2600d3f008784be29531b6ee73ee6e02 P dc3b1d8f08084d9aaac8bfaaa429399f: Deleting consensus metadata
I20260925 19:09:57.412909 2546 catalog_manager.cc:4841] TS b6949d1218204e7092d266abbbca6b13 (127.27.174.66:36049): tablet 2600d3f008784be29531b6ee73ee6e02 (table test-table [id=ba64b09a024c4cf08927e902ceb45c0a]) successfully deleted
I20260925 19:09:57.413240 2546 catalog_manager.cc:4841] TS dc3b1d8f08084d9aaac8bfaaa429399f (127.27.174.67:37405): tablet 2600d3f008784be29531b6ee73ee6e02 (table test-table [id=ba64b09a024c4cf08927e902ceb45c0a]) successfully deleted
I20260925 19:09:57.416465 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:57.416386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38289 (local address 127.0.0.1:57532)
0925 19:09:57.416415 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:57.416440 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 19:09:57.416595 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2932
W20260925 19:09:57.416620 2544 catalog_manager.cc:4584] TS 832df0ab2375467da11a100246364c39 (127.27.174.65:38289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2600d3f008784be29531b6ee73ee6e02: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38289: connect: Connection refused (error 111)
I20260925 19:09:57.422575 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 3068
I20260925 19:09:57.427197 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2425
I20260925 19:09:57.431831 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 2613
2026-09-25T19:09:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6467 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:57 dist-test-slave-fcdn krb5kdc[3326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:57 dist-test-slave-fcdn krb5kdc[3326](info): set up 2 sockets
Sep 25 19:09:57 dist-test-slave-fcdn krb5kdc[3326](info): commencing operation
krb5kdc: starting...
W20260925 19:09:58.484198 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:09:58 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
I20260925 19:09:58.603116 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33067
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:58.684888 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:58.685103 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:58.685127 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:58.686241 3339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:58.686278 3339 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:09:58.686291 3339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:58.686304 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:58.686317 3339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:58.686329 3339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:58.687891 3339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33067
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:58.688088 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:58.688265 3339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:58.693132 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:58.693132 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:58.693127 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:58.693256 3339 server_base.cc:1034] running on GCE node
I20260925 19:09:58.693704 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:58.694000 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:58.695169 3339 hybrid_clock.cc:648] HybridClock initialized: now 1790363398695157 us; error 33 us; skew 500 ppm
Sep 25 19:09:58 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:58.701894 3339 init.cc:377] Logged in from keytab as oryx/127.27.174.126@KRBTEST.COM (short username oryx)
I20260925 19:09:58.702375 3339 webserver.cc:460] Webserver started at http://127.27.174.126:43377/ using document root <none> and password file <none>
I20260925 19:09:58.702632 3339 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:58.702701 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:58.702858 3339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:58.704011 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "a7141cc4084d4d0ebe689926053a4d6d"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "6c54e33ddf32d35458fd3e1238c62a67"
server_key_iv: "e48112ce0b51257390ec50ecf54044ff"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.704373 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "a7141cc4084d4d0ebe689926053a4d6d"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "6c54e33ddf32d35458fd3e1238c62a67"
server_key_iv: "e48112ce0b51257390ec50ecf54044ff"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.705727 3339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:58.706462 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:58.706670 3339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:58.706771 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "a7141cc4084d4d0ebe689926053a4d6d"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "6c54e33ddf32d35458fd3e1238c62a67"
server_key_iv: "e48112ce0b51257390ec50ecf54044ff"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.706848 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:58.721297 3339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:58.723384 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:58.723549 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:58.728072 3339 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33067
I20260925 19:09:58.728149 3416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33067 every 8 connection(s)
I20260925 19:09:58.728478 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:58.729199 3417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:58.731455 3417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Bootstrap starting.
I20260925 19:09:58.732450 3417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:58.732828 3417 log.cc:824] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:58.733563 3417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: No bootstrap required, opened a new log
I20260925 19:09:58.735694 3417 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } }
I20260925 19:09:58.735834 3417 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:58.735857 3417 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7141cc4084d4d0ebe689926053a4d6d, State: Initialized, Role: FOLLOWER
I20260925 19:09:58.736018 3417 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } }
I20260925 19:09:58.736093 3417 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:58.736115 3417 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:58.736182 3417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:58.736697 3417 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } }
I20260925 19:09:58.736804 3417 leader_election.cc:302] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7141cc4084d4d0ebe689926053a4d6d; no voters:
I20260925 19:09:58.736977 3417 leader_election.cc:288] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:58.737049 3420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:58.737239 3420 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [term 1 LEADER]: Becoming Leader. State: Replica: a7141cc4084d4d0ebe689926053a4d6d, State: Running, Role: LEADER
I20260925 19:09:58.737248 3417 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:58.737344 3420 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } }
I20260925 19:09:58.737738 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7141cc4084d4d0ebe689926053a4d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } } }
I20260925 19:09:58.737895 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 3339
I20260925 19:09:58.737898 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:58.737974 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:09:58.738197 28345 external_mini_cluster.cc:1428] Setting key 467ec917f518f97e72d7143812ec004d
I20260925 19:09:58.738746 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7141cc4084d4d0ebe689926053a4d6d. Latest consensus state: current_term: 1 leader_uuid: "a7141cc4084d4d0ebe689926053a4d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7141cc4084d4d0ebe689926053a4d6d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33067 } } }
I20260925 19:09:58.738826 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:58.738948 3429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:58.739513 3429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:58.741032 3429 catalog_manager.cc:1347] Generated new cluster ID: 9b30eb3d8a3a4469b438fd97c0bbcea0
I20260925 19:09:58.741093 3429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:58 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:58.748651 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:58.738823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.27.174.126:33067)
0925 19:09:58.739070 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:58.739081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:58.739117 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:09:58.739258 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:58.739266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:58.739321 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:58.739468 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:58.740156 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.741493 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.742276 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.742716 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.742834 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:58.745233 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:58.745237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:58.745237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:58.745249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:58.747072 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.747448 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.747450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.747451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.747547 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.747790 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.747791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.747792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.748046 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:58.748221 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:58.748297 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 19:09:58.748434 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":146,"threads_started":1}
I20260925 19:09:58.750288 3429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:58.750762 3429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:58.754264 3429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Generated new TSK 0
I20260925 19:09:58.754469 3429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
I20260925 19:09:58.782383 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33067
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:58.864221 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:58.864420 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:58.864444 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:58.865612 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:58.865653 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:58.865672 3443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:58.865690 3443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:09:58.867270 3443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:33067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:58.867523 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:58.867789 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:58.872777 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:58.872810 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:58.872808 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:58.873001 3443 server_base.cc:1034] running on GCE node
I20260925 19:09:58.873263 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:58.873457 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:58.874600 3443 hybrid_clock.cc:648] HybridClock initialized: now 1790363398874577 us; error 40 us; skew 500 ppm
Sep 25 19:09:58 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:58.877262 3443 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20260925 19:09:58.877727 3443 webserver.cc:460] Webserver started at http://127.27.174.65:46443/ using document root <none> and password file <none>
I20260925 19:09:58.877964 3443 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:58.878031 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:58.878183 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:58.879238 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "a8297d071cc04dacac406c94e51c25cc"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "fa29bb25256ada71d277877d28f1fa32"
server_key_iv: "65c1bfcce4e1b8ba9ad1b30a8cf6d27f"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.879669 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "a8297d071cc04dacac406c94e51c25cc"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "fa29bb25256ada71d277877d28f1fa32"
server_key_iv: "65c1bfcce4e1b8ba9ad1b30a8cf6d27f"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.881507 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 19:09:58.882453 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:58.882831 3443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:58.882920 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "a8297d071cc04dacac406c94e51c25cc"
format_stamp: "Formatted at 2026-09-25 19:09:58 on dist-test-slave-fcdn"
server_key: "fa29bb25256ada71d277877d28f1fa32"
server_key_iv: "65c1bfcce4e1b8ba9ad1b30a8cf6d27f"
server_key_version: "encryptionkey@0"
I20260925 19:09:58.882993 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:58.892620 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:58.894899 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:58.895049 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:58.895550 3443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:58.895617 3443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:58.895673 3443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:58.895699 3443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:58.901997 3443 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35049
I20260925 19:09:58.902146 3573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35049 every 8 connection(s)
I20260925 19:09:58.902406 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:09:58 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:58.906075 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:58.896019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47463 (local address 127.27.174.126:33067)
0925 19:09:58.896079 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:58.896081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:58.896366 (+ 285us) server_negotiation.cc:407] Connection header received
0925 19:09:58.896961 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:58.896963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:58.896980 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:58.897062 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:58.897925 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.898795 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.900389 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.900763 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.900850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:58.903399 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:58.903400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:58.903401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:58.903408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:58.904690 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.905110 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.905111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.905112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.905177 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.905426 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.905427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.905427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.905551 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:58.905631 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:58.905901 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 19:09:58.905983 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:58.906267 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:58.896005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.65:47463)
0925 19:09:58.896259 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:58.896287 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:58.896778 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:58.897160 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:58.897165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:58.897508 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:58.897814 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:58.897821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.898892 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.898893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:58.900295 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:58.900297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.900859 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.900860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:58.901007 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:58.901633 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:58.901641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:58.903296 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:58.904779 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:58.904786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:58.904788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:58.905018 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:58.905245 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:58.905245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:58.905246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:58.905356 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:58.905623 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:58.905629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:58.905771 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 19:09:58.905880 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:09:58.906042 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260925 19:09:58.906761 3574 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33067
I20260925 19:09:58.906874 3574 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:58.907079 3574 heartbeater.cc:511] Master 127.27.174.126:33067 requested a full tablet report, sending...
I20260925 19:09:58.907153 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 3443
I20260925 19:09:58.907248 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:09:58.907461 28345 external_mini_cluster.cc:1428] Setting key d003910f0f40f05bf85dad5702dbd018
I20260925 19:09:58.907750 3370 ts_manager.cc:194] Registered new tserver with Master: a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049)
I20260925 19:09:58.908630 3370 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:47463
I20260925 19:09:58.914289 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:58.907862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.65:35847)
0925 19:09:58.908043 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:58.908054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:58.908131 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:58.908276 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:58.908276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:58.908443 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:58.908490 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:58.908491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.909392 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.909393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:58.910323 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:58.910326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.910754 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.910755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:58.910830 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:58.911201 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:58.911206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:58.911591 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:58.913367 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:58.913367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:58.913368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:58.913522 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:58.913740 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:58.913741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:58.913741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:58.913797 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:58.914078 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:58.914079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:58.914112 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 19:09:58.914162 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:09:58.914230 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260925 19:09:58.914371 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:58.907910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35847 (local address 127.27.174.126:33067)
0925 19:09:58.907953 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:58.907955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:58.908105 (+ 150us) server_negotiation.cc:407] Connection header received
0925 19:09:58.908208 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:58.908208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:58.908218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:58.908263 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:58.908583 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.909297 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.910418 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:58.910685 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:58.910753 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:58.911714 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:58.911715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:58.911715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:58.911721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:58.913281 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.913607 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.913608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.913608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.913671 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:58.913859 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:58.913860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:58.913860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:58.914002 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:58.914080 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:58.914191 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:09:58.914274 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
I20260925 19:09:58.933900 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33067
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:59.013536 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:59.013726 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:59.013749 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:59.014889 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:59.014928 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:59.014950 3583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:59.014966 3583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:09:59.016453 3583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:33067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:59.016685 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:59.016901 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:59.021826 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:59.021860 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:59.021983 3583 server_base.cc:1034] running on GCE node
W20260925 19:09:59.021823 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:59.022248 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:59.022477 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:59.023669 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790363399023642 us; error 39 us; skew 500 ppm
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:59.026294 3583 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20260925 19:09:59.026775 3583 webserver.cc:460] Webserver started at http://127.27.174.66:45995/ using document root <none> and password file <none>
I20260925 19:09:59.026999 3583 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:59.027074 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:59.027222 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:59.028287 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "451f4125b8b4ebf55ac5211aeff7d097"
server_key_iv: "ac6398dfcafe2d68b65fb6a01cb93e32"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.028640 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "451f4125b8b4ebf55ac5211aeff7d097"
server_key_iv: "ac6398dfcafe2d68b65fb6a01cb93e32"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.030005 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:59.030789 3598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.030969 3583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.031083 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "451f4125b8b4ebf55ac5211aeff7d097"
server_key_iv: "ac6398dfcafe2d68b65fb6a01cb93e32"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.031158 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:59.063433 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:59.065977 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:59.066150 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:59.066679 3583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:59.066745 3583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.066790 3583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:59.066820 3583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:59.074031 3583 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:46339
I20260925 19:09:59.074122 3713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:46339 every 8 connection(s)
I20260925 19:09:59.074432 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:09:59.077138 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.067265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39633 (local address 127.27.174.126:33067)
0925 19:09:59.067320 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.067322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.067578 (+ 256us) server_negotiation.cc:407] Connection header received
0925 19:09:59.068160 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.068161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.068174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.068235 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.069159 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.069974 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.071677 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.071895 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.071978 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.074271 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.074273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.074273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.074281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.075940 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.076304 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.076305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.076305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.076374 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.076637 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.076638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.076638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.076740 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.076791 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.077011 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.077074 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:09:59.077404 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.067263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.66:39633)
0925 19:09:59.067424 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.067458 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.068049 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.068308 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.068313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.068729 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:59.069045 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.069052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.070081 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.070083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.071553 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.071556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.072000 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.072001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.072141 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.072711 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:59.072721 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:59.074138 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:59.076011 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.076015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.076017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.076232 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.076441 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.076442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.076442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.076578 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.076794 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:59.076798 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:59.076907 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.076988 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.077147 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260925 19:09:59.077909 3714 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33067
I20260925 19:09:59.077991 3714 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:59.078173 3714 heartbeater.cc:511] Master 127.27.174.126:33067 requested a full tablet report, sending...
I20260925 19:09:59.078565 3371 ts_manager.cc:194] Registered new tserver with Master: 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.078632 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 3583
I20260925 19:09:59.078727 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:09:59.078896 28345 external_mini_cluster.cc:1428] Setting key 6f356b0f929ec1df70ef0b30c5ddfabd
I20260925 19:09:59.079082 3371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:39633
I20260925 19:09:59.084975 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.079140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:60131 (local address 127.27.174.126:33067)
0925 19:09:59.079183 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.079185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.079194 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.079292 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.079293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.079321 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.079370 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.079726 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.080186 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.080880 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.081090 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.081169 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.082072 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.082074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.082074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.082081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.083671 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.084093 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.084094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.084095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.084156 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.084380 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.084381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.084382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.084537 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.084620 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.084787 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.084866 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:59.084981 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.079008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.66:60131)
0925 19:09:59.079145 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.079156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.079208 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.079373 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.079373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.079556 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:59.079659 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.079661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.080280 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.080281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.080800 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.080801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.081171 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.081172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.081238 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.081672 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:59.081677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:59.081948 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:59.083792 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.083793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.083793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.083991 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.084235 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.084236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.084236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.084299 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.084611 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:59.084612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:59.084662 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.084760 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.084868 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
I20260925 19:09:59.106746 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33067
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:59.190265 3723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:59.190469 3723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:59.190493 3723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:59.191847 3723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:59.191888 3723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:59.191908 3723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:09:59.191946 3723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:09:59.193452 3723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:33067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:59.193708 3723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:59.193935 3723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:59.198748 3729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:59.198894 3731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:59.198745 3728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:59.199064 3723 server_base.cc:1034] running on GCE node
I20260925 19:09:59.199425 3723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:59.199750 3723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:59.200905 3723 hybrid_clock.cc:648] HybridClock initialized: now 1790363399200891 us; error 33 us; skew 500 ppm
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:59.203781 3723 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20260925 19:09:59.204244 3723 webserver.cc:460] Webserver started at http://127.27.174.67:33417/ using document root <none> and password file <none>
I20260925 19:09:59.204454 3723 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:59.204495 3723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:59.204581 3723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:59.205538 3723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "e49c54501e714bdba43d279254c173a3"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "fbb2904d1e7400c0ace7b7680be13c99"
server_key_iv: "0a2ef8efaf585ed3fa3242e60adf4210"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.205838 3723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "e49c54501e714bdba43d279254c173a3"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "fbb2904d1e7400c0ace7b7680be13c99"
server_key_iv: "0a2ef8efaf585ed3fa3242e60adf4210"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.207295 3723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:59.208094 3738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.208240 3723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.208312 3723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "e49c54501e714bdba43d279254c173a3"
format_stamp: "Formatted at 2026-09-25 19:09:59 on dist-test-slave-fcdn"
server_key: "fbb2904d1e7400c0ace7b7680be13c99"
server_key_iv: "0a2ef8efaf585ed3fa3242e60adf4210"
server_key_version: "encryptionkey@0"
I20260925 19:09:59.208379 3723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:59.226310 3723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:59.228657 3723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:59.228807 3723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:59.229415 3723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:09:59.229460 3723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.229496 3723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:09:59.229529 3723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:59.237399 3723 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:33403
I20260925 19:09:59.237483 3853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:33403 every 8 connection(s)
I20260925 19:09:59.237803 3723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:09:59.240389 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.229816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:58965 (local address 127.27.174.126:33067)
0925 19:09:59.229867 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.229869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.230148 (+ 279us) server_negotiation.cc:407] Connection header received
0925 19:09:59.230866 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.230868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.230881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.230952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.231892 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.232794 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.234307 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.234515 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.234590 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.237297 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.237298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.237298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.237306 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.238969 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.239351 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.239353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.239353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.239418 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.239709 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.239710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.239710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.239831 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.239956 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.240224 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.240303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:59.240546 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.229833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.67:58965)
0925 19:09:59.229991 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.230043 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.230747 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.231034 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.231039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.231446 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:59.231798 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.231805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.232914 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.232916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.234198 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.234201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.234608 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.234608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.234717 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.235226 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:59.235236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:59.237183 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:59.239058 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.239064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.239066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.239275 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.239506 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.239507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.239507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.239654 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.239949 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:59.239954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:59.240092 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.240189 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.240331 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260925 19:09:59.240998 3854 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33067
I20260925 19:09:59.241076 3854 heartbeater.cc:464] Registering TS with master...
I20260925 19:09:59.241245 3854 heartbeater.cc:511] Master 127.27.174.126:33067 requested a full tablet report, sending...
I20260925 19:09:59.241582 3371 ts_manager.cc:194] Registered new tserver with Master: e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403)
I20260925 19:09:59.241881 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 3723
I20260925 19:09:59.241966 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:09:59.242101 3371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:58965
I20260925 19:09:59.242170 28345 external_mini_cluster.cc:1428] Setting key d198ba67345e2aea86cd9d4221cb16b3
I20260925 19:09:59.243006 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:09:59.248257 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.241972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.27.174.67:39519)
0925 19:09:59.242124 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.242132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.242200 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.242357 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.242358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.242563 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:59.242663 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.242665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.243412 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.243413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.243998 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.244000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.244375 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.244375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.244446 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.245025 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:59.245040 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:59.245452 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:59.247211 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.247212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.247212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.247427 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.247641 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.247641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.247642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.247705 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.248010 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:59.248011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:59.248056 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.248113 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.248185 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260925 19:09:59.248399 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.242018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:39519 (local address 127.27.174.126:33067)
0925 19:09:59.242056 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.242058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.242188 (+ 130us) server_negotiation.cc:407] Connection header received
0925 19:09:59.242258 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.242259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.242268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.242316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.242730 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.243324 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.244099 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.244296 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.244374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.245576 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.245578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.245578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.245583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.247125 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.247504 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.247504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.247505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.247554 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.247768 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.247768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.247768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.247927 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.248012 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.248163 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.248243 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:09:59.264786 3433 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:59.259656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.27.174.126:33067)
0925 19:09:59.259695 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.259699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.259711 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:59.259826 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.259827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.259837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.259887 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.260377 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.260859 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.261571 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.261753 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.261803 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.264711 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44386: BlockingRecv error: recv got EOF from 127.0.0.1:44386 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:59.271778 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.265548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.27.174.126:33067)
0925 19:09:59.265608 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.265610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.265703 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:09:59.265740 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.265741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.265748 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.265798 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.266268 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.266726 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.267380 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.267544 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.267694 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.269556 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.269557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.269558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.269564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.270750 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.271046 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.271047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.271047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.271094 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.271249 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.271249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.271250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.271341 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.271413 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.271560 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.271682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:09:59.273442 3371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:59.274957 3371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:59.281387 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:46339 (local address 127.0.0.1:46984)
0925 19:09:59.277424 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.277436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.277491 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.277825 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.277828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.277831 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.277902 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.277904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.278906 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.278907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.280384 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.280386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.281067 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.281163 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281177 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.281245 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.281293 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":76,"threads_started":1}
I20260925 19:09:59.281562 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35049 (local address 127.0.0.1:35670)
0925 19:09:59.277682 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.277693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.277744 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.278005 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.278005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.278007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.278041 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.278043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.279317 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.279318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.280323 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.280326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.281309 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.281377 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281381 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.281428 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.281465 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":60,"threads_started":1}
I20260925 19:09:59.281690 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.27.174.67:33403)
0925 19:09:59.277680 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.277683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.277712 (+ 29us) server_negotiation.cc:407] Connection header received
0925 19:09:59.277756 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.277758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.277772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.277918 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.278171 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.279207 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.280478 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.281183 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281245 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281306 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.281371 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.281416 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":101,"threads_started":1}
I20260925 19:09:59.281728 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:33403 (local address 127.0.0.1:60472)
0925 19:09:59.277288 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.277330 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.277393 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.277918 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.277919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.277921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.278043 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.278044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.279305 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.279306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.280415 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.280417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.281245 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.281297 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281300 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.281358 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.281386 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":32,"thread_start_us":43,"threads_started":1}
I20260925 19:09:59.281817 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.27.174.65:35049)
0925 19:09:59.277622 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.277628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.277808 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:09:59.277851 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.277853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.277870 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.277934 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.278122 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.279231 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.280404 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.281139 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281214 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281272 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.281676 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.281722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 19:09:59.281780 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.277484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.27.174.66:46339)
0925 19:09:59.277652 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.277656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.277669 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:09:59.277715 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.277717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.277733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.277814 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.277987 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.278838 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.280478 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.280979 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.281069 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.281152 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.281578 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.281619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 19:09:59.282588 3648 tablet_service.cc:1467] Processing CreateTablet for tablet e670ad5d391a492eb3073df37faea368 (DEFAULT_TABLE table=test-table [id=097e454e45e846a7b0b1fbee509ec4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.282589 3788 tablet_service.cc:1467] Processing CreateTablet for tablet e670ad5d391a492eb3073df37faea368 (DEFAULT_TABLE table=test-table [id=097e454e45e846a7b0b1fbee509ec4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.282928 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670ad5d391a492eb3073df37faea368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.282932 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670ad5d391a492eb3073df37faea368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.284119 3508 tablet_service.cc:1467] Processing CreateTablet for tablet e670ad5d391a492eb3073df37faea368 (DEFAULT_TABLE table=test-table [id=097e454e45e846a7b0b1fbee509ec4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.284472 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670ad5d391a492eb3073df37faea368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.286062 3882 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Bootstrap starting.
I20260925 19:09:59.286732 3882 tablet_bootstrap.cc:654] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.286949 3883 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Bootstrap starting.
I20260925 19:09:59.287109 3882 log.cc:824] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:59.287511 3883 tablet_bootstrap.cc:654] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.287850 3883 log.cc:824] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:59.288025 3882 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: No bootstrap required, opened a new log
I20260925 19:09:59.288087 3882 ts_tablet_manager.cc:1397] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:59.288654 3883 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: No bootstrap required, opened a new log
I20260925 19:09:59.288724 3883 ts_tablet_manager.cc:1397] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:09:59.290422 3882 raft_consensus.cc:348] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.290606 3882 raft_consensus.cc:374] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.290644 3882 raft_consensus.cc:729] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c54501e714bdba43d279254c173a3, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.290658 3883 raft_consensus.cc:348] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.290794 3883 raft_consensus.cc:374] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.290761 3882 consensus_queue.cc:260] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.290833 3883 raft_consensus.cc:729] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.290940 3883 consensus_queue.cc:260] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.291028 3882 ts_tablet_manager.cc:1428] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:09:59.291072 3854 heartbeater.cc:503] Master 127.27.174.126:33067 was elected leader, sending a full tablet report...
I20260925 19:09:59.291193 3883 ts_tablet_manager.cc:1428] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 19:09:59.291247 3714 heartbeater.cc:503] Master 127.27.174.126:33067 was elected leader, sending a full tablet report...
I20260925 19:09:59.291746 3886 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Bootstrap starting.
I20260925 19:09:59.292420 3886 tablet_bootstrap.cc:654] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.292716 3886 log.cc:824] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:59.293458 3886 tablet_bootstrap.cc:492] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: No bootstrap required, opened a new log
I20260925 19:09:59.293540 3886 ts_tablet_manager.cc:1397] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:59.294910 3886 raft_consensus.cc:348] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.295058 3886 raft_consensus.cc:374] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.295097 3886 raft_consensus.cc:729] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8297d071cc04dacac406c94e51c25cc, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.295210 3886 consensus_queue.cc:260] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.295445 3886 ts_tablet_manager.cc:1428] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:09:59.295478 3574 heartbeater.cc:503] Master 127.27.174.126:33067 was elected leader, sending a full tablet report...
W20260925 19:09:59.325052 3715 tablet.cc:2369] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:59.382406 3888 raft_consensus.cc:482] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:59.382607 3888 raft_consensus.cc:504] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.383018 3888 leader_election.cc:288] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403), a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049)
I20260925 19:09:59.386932 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.383293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57797 (local address 127.27.174.65:35049)
0925 19:09:59.383340 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.383342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.383354 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:09:59.383430 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.383431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.383441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.383494 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.383701 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.384477 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.385955 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.386542 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.386632 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.386641 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.386822 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.386862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:59.386932 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.383302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38349 (local address 127.27.174.67:33403)
0925 19:09:59.383357 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.383359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.383584 (+ 225us) server_negotiation.cc:407] Connection header received
0925 19:09:59.383669 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.383670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.383680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.383735 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.383883 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.384569 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.385955 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.386552 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.386622 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.386632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.386822 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.386862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:59.386952 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.383239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:33403 (local address 127.27.174.66:38349)
0925 19:09:59.383503 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.383513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.383571 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.383753 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.383753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.383763 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.383816 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.383817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.384636 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.384637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.385862 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.385865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.386634 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.386634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.386727 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.386731 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.386799 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.386850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":88,"thread_start_us":114,"threads_started":1}
I20260925 19:09:59.386987 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.383235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35049 (local address 127.27.174.66:57797)
0925 19:09:59.383288 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.383301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.383349 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.383527 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.383528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.383530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.383628 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.383630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.384572 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.384573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.385862 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.385865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.386633 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.386634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.386729 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.386731 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.386799 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.386850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":15}
I20260925 19:09:59.387521 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670ad5d391a492eb3073df37faea368" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc" is_pre_election: true
I20260925 19:09:59.387542 3808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670ad5d391a492eb3073df37faea368" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c54501e714bdba43d279254c173a3" is_pre_election: true
I20260925 19:09:59.387727 3528 raft_consensus.cc:2454] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 0.
I20260925 19:09:59.387727 3808 raft_consensus.cc:2454] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 0.
I20260925 19:09:59.387975 3602 leader_election.cc:302] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.388131 3888 raft_consensus.cc:2781] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:59.388192 3888 raft_consensus.cc:482] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:59.388216 3888 raft_consensus.cc:3037] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.389155 3888 raft_consensus.cc:504] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.389299 3888 leader_election.cc:288] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 election: Requested vote from peers e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403), a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049)
I20260925 19:09:59.389482 3808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670ad5d391a492eb3073df37faea368" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c54501e714bdba43d279254c173a3"
I20260925 19:09:59.389472 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670ad5d391a492eb3073df37faea368" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc"
I20260925 19:09:59.389557 3528 raft_consensus.cc:3037] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.389557 3808 raft_consensus.cc:3037] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.390427 3528 raft_consensus.cc:2454] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 1.
I20260925 19:09:59.390427 3808 raft_consensus.cc:2454] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 1.
I20260925 19:09:59.390630 3602 leader_election.cc:302] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.390734 3888 raft_consensus.cc:2781] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:59.390884 3888 raft_consensus.cc:686] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Running, Role: LEADER
I20260925 19:09:59.390983 3888 consensus_queue.cc:237] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } }
I20260925 19:09:59.391858 3371 catalog_manager.cc:5495] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } health_report { overall_health: UNKNOWN } } }
W20260925 19:09:59.403167 3575 tablet.cc:2369] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:59.425173 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.422862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.27.174.66:46339)
0925 19:09:59.422921 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.422923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.422932 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.422977 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.422978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.422994 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.423052 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.423199 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.423721 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.424535 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.424715 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.424764 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.424915 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.425004 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.425060 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.425101 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.425117 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:09:59.427029 3808 raft_consensus.cc:1261] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Refusing update from remote peer 7259cb25fc2c4012bc43a5e7762aa5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.427028 3528 raft_consensus.cc:1261] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Refusing update from remote peer 7259cb25fc2c4012bc43a5e7762aa5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.427423 3888 consensus_queue.cc:1035] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.427606 3892 consensus_queue.cc:1035] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.430450 3898 mvcc.cc:205] Tried to move back new op lower bound from 7333328484050157568 to 7333328483906015232. Current Snapshot: MvccSnapshot[applied={T|T < 7333328484050157568}]
I20260925 19:09:59.432063 3370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.432157 3370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.432916 3370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Sending DeleteTablet for 3 replicas of tablet e670ad5d391a492eb3073df37faea368
I20260925 19:09:59.433204 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet e670ad5d391a492eb3073df37faea368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60472
I20260925 19:09:59.433296 3648 tablet_service.cc:1514] Processing DeleteTablet for tablet e670ad5d391a492eb3073df37faea368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46984
I20260925 19:09:59.433339 3508 tablet_service.cc:1514] Processing DeleteTablet for tablet e670ad5d391a492eb3073df37faea368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35670
I20260925 19:09:59.433533 3904 tablet_replica.cc:330] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: stopping tablet replica
I20260925 19:09:59.433627 3904 raft_consensus.cc:2229] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.433578 3370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:59.433729 3370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:59.433985 3904 raft_consensus.cc:2258] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.434500 3904 ts_tablet_manager.cc:1905] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.434733 3906 tablet_replica.cc:330] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: stopping tablet replica
I20260925 19:09:59.434818 3906 raft_consensus.cc:2229] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.434899 3906 raft_consensus.cc:2258] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.435034 3905 tablet_replica.cc:330] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: stopping tablet replica
I20260925 19:09:59.435189 3788 tablet_service.cc:1467] Processing CreateTablet for tablet d24866505db14ca2af9d82e6e025bb29 (DEFAULT_TABLE table=test-table [id=af1f4ced7d9944498879da78d76153a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.435238 3508 tablet_service.cc:1467] Processing CreateTablet for tablet d24866505db14ca2af9d82e6e025bb29 (DEFAULT_TABLE table=test-table [id=af1f4ced7d9944498879da78d76153a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.435283 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24866505db14ca2af9d82e6e025bb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.435364 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24866505db14ca2af9d82e6e025bb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.436230 3904 ts_tablet_manager.cc:1918] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.436296 3904 log.cc:1196] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/e670ad5d391a492eb3073df37faea368
I20260925 19:09:59.436540 3904 ts_tablet_manager.cc:1939] T e670ad5d391a492eb3073df37faea368 P e49c54501e714bdba43d279254c173a3: Deleting consensus metadata
I20260925 19:09:59.436664 3882 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Bootstrap starting.
I20260925 19:09:59.437036 3882 tablet_bootstrap.cc:654] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.437165 3886 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Bootstrap starting.
I20260925 19:09:59.437290 3358 catalog_manager.cc:4841] TS e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403): tablet e670ad5d391a492eb3073df37faea368 (table test-table [id=097e454e45e846a7b0b1fbee509ec4a1]) successfully deleted
I20260925 19:09:59.437553 3886 tablet_bootstrap.cc:654] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.435110 3905 raft_consensus.cc:2229] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:59.435714 3648 tablet_service.cc:1467] Processing CreateTablet for tablet d24866505db14ca2af9d82e6e025bb29 (DEFAULT_TABLE table=test-table [id=af1f4ced7d9944498879da78d76153a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.437805 3905 raft_consensus.cc:2258] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.437822 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24866505db14ca2af9d82e6e025bb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.438346 3882 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: No bootstrap required, opened a new log
I20260925 19:09:59.438346 3886 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: No bootstrap required, opened a new log
I20260925 19:09:59.438386 3882 ts_tablet_manager.cc:1397] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:59.438386 3886 ts_tablet_manager.cc:1397] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.438529 3882 raft_consensus.cc:348] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.438587 3882 raft_consensus.cc:374] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.438546 3886 raft_consensus.cc:348] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.438608 3882 raft_consensus.cc:729] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c54501e714bdba43d279254c173a3, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.438616 3886 raft_consensus.cc:374] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.438654 3886 raft_consensus.cc:729] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8297d071cc04dacac406c94e51c25cc, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.438648 3882 consensus_queue.cc:260] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.438692 3886 consensus_queue.cc:260] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.438736 3882 ts_tablet_manager.cc:1428] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.438788 3886 ts_tablet_manager.cc:1428] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.439483 3883 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Bootstrap starting.
I20260925 19:09:59.439720 3905 ts_tablet_manager.cc:1905] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.439983 3883 tablet_bootstrap.cc:654] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.440547 3906 ts_tablet_manager.cc:1905] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.441068 3883 tablet_bootstrap.cc:492] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: No bootstrap required, opened a new log
I20260925 19:09:59.441120 3883 ts_tablet_manager.cc:1397] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:09:59.441215 3905 ts_tablet_manager.cc:1918] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.441274 3905 log.cc:1196] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/e670ad5d391a492eb3073df37faea368
I20260925 19:09:59.441267 3883 raft_consensus.cc:348] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.441347 3883 raft_consensus.cc:374] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.441371 3883 raft_consensus.cc:729] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.441416 3883 consensus_queue.cc:260] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.441501 3883 ts_tablet_manager.cc:1428] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:59.441557 3905 ts_tablet_manager.cc:1939] T e670ad5d391a492eb3073df37faea368 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting consensus metadata
I20260925 19:09:59.441917 3355 catalog_manager.cc:4841] TS 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339): tablet e670ad5d391a492eb3073df37faea368 (table test-table [id=097e454e45e846a7b0b1fbee509ec4a1]) successfully deleted
I20260925 19:09:59.442384 3906 ts_tablet_manager.cc:1918] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.442471 3906 log.cc:1196] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/e670ad5d391a492eb3073df37faea368
I20260925 19:09:59.442688 3906 ts_tablet_manager.cc:1939] T e670ad5d391a492eb3073df37faea368 P a8297d071cc04dacac406c94e51c25cc: Deleting consensus metadata
I20260925 19:09:59.443222 3355 catalog_manager.cc:4841] TS a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049): tablet e670ad5d391a492eb3073df37faea368 (table test-table [id=097e454e45e846a7b0b1fbee509ec4a1]) successfully deleted
W20260925 19:09:59.488674 3855 tablet.cc:2369] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:09:59.554536 3887 raft_consensus.cc:482] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:59.554632 3887 raft_consensus.cc:504] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.554996 3887 leader_election.cc:288] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.559012 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.555300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:46019 (local address 127.27.174.65:35049)
0925 19:09:59.555360 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.555362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.555371 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.555402 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.555403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.555411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.555537 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.555732 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.556506 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558073 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.558630 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558704 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.558712 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.558909 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.558948 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:09:59.559012 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.555391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:39655 (local address 127.27.174.66:46339)
0925 19:09:59.555436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.555439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.555483 (+ 44us) server_negotiation.cc:407] Connection header received
0925 19:09:59.555565 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.555566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.555582 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.555671 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:09:59.555836 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.556513 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558074 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.558626 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.558712 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.558909 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.558944 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:59.559029 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35049 (local address 127.27.174.67:46019)
0925 19:09:59.555264 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.555278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.555341 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.555544 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.555545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.555546 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.555648 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.555651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.556625 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.556626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.557969 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.557970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.558726 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.558820 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.558825 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.558888 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.558933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":124}
I20260925 19:09:59.559043 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.555314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:46339 (local address 127.27.174.67:39655)
0925 19:09:59.555428 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.555438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.555496 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.555680 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.555681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.555683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:09:59.555769 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.555770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.556624 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.556626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.557966 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.557969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.558747 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.558748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.558820 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.558825 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.558888 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.558933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":43,"thread_start_us":61,"threads_started":1}
I20260925 19:09:59.559340 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24866505db14ca2af9d82e6e025bb29" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" is_pre_election: true
I20260925 19:09:59.559422 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24866505db14ca2af9d82e6e025bb29" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc" is_pre_election: true
I20260925 19:09:59.559494 3528 raft_consensus.cc:2454] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.559489 3668 raft_consensus.cc:2454] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.559677 3739 leader_election.cc:302] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.559844 3887 raft_consensus.cc:2781] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:59.559901 3887 raft_consensus.cc:482] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:59.559930 3887 raft_consensus.cc:3037] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.560761 3887 raft_consensus.cc:504] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.560909 3887 leader_election.cc:288] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Requested vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.561108 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24866505db14ca2af9d82e6e025bb29" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc"
I20260925 19:09:59.561194 3528 raft_consensus.cc:3037] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.561163 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24866505db14ca2af9d82e6e025bb29" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
I20260925 19:09:59.561255 3668 raft_consensus.cc:3037] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.561815 3528 raft_consensus.cc:2454] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.561913 3668 raft_consensus.cc:2454] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.561995 3739 leader_election.cc:302] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8297d071cc04dacac406c94e51c25cc, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.562109 3887 raft_consensus.cc:2781] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:59.562284 3887 raft_consensus.cc:686] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 1 LEADER]: Becoming Leader. State: Replica: e49c54501e714bdba43d279254c173a3, State: Running, Role: LEADER
I20260925 19:09:59.562371 3887 consensus_queue.cc:237] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.563151 3370 catalog_manager.cc:5495] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49c54501e714bdba43d279254c173a3 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } health_report { overall_health: HEALTHY } } }
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:59.611338 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.604252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33067 (local address 127.0.0.1:44404)
0925 19:09:59.604429 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.604440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.604532 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.604691 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.604692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.605004 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:59.605048 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.605050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.605716 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.605717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.606298 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.606299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.606648 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.606649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.606698 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.607230 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:59.607234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:59.608612 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:59.610120 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.610125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.610127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.610327 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.610629 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:59.610630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:59.610630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:59.610720 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:59.610976 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:59.610982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:59.611025 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.611118 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.611263 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260925 19:09:59.611461 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.604322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.27.174.126:33067)
0925 19:09:59.604401 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.604403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.604513 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:09:59.604599 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.604600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.604616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.604707 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.605150 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.605636 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.606388 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.606564 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.606649 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.608689 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.608690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.608690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.608697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.610037 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.610410 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.610411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.610411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.610475 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.610776 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.610777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.610777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.610908 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.610967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.611138 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.611309 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":51}
I20260925 19:09:59.613406 3370 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:44404:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:09:59.613633 3370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:59.615157 3648 tablet_service.cc:1467] Processing CreateTablet for tablet 071414cbd6d94f4b815a7562035120c8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8937d35dc89c4ccbb6d59905adaa8476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:59.615147 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 071414cbd6d94f4b815a7562035120c8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8937d35dc89c4ccbb6d59905adaa8476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:59.615302 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071414cbd6d94f4b815a7562035120c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.615301 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071414cbd6d94f4b815a7562035120c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.615254 3508 tablet_service.cc:1467] Processing CreateTablet for tablet 071414cbd6d94f4b815a7562035120c8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8937d35dc89c4ccbb6d59905adaa8476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:09:59.615389 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071414cbd6d94f4b815a7562035120c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.616555 3886 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: Bootstrap starting.
I20260925 19:09:59.616817 3883 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: Bootstrap starting.
I20260925 19:09:59.616820 3882 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: Bootstrap starting.
I20260925 19:09:59.617033 3886 tablet_bootstrap.cc:654] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.617100 3883 tablet_bootstrap.cc:654] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.617215 3882 tablet_bootstrap.cc:654] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.617805 3883 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: No bootstrap required, opened a new log
I20260925 19:09:59.617864 3883 ts_tablet_manager.cc:1397] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:59.617888 3886 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: No bootstrap required, opened a new log
I20260925 19:09:59.617930 3886 ts_tablet_manager.cc:1397] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.617986 3883 raft_consensus.cc:348] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.618065 3883 raft_consensus.cc:374] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.618086 3883 raft_consensus.cc:729] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.618085 3886 raft_consensus.cc:348] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.618130 3883 consensus_queue.cc:260] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.618162 3886 raft_consensus.cc:374] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.618187 3886 raft_consensus.cc:729] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8297d071cc04dacac406c94e51c25cc, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.618232 3883 ts_tablet_manager.cc:1428] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:59.618239 3886 consensus_queue.cc:260] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.618327 3886 ts_tablet_manager.cc:1428] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:59.618345 3888 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.618420 3888 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.618455 3890 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.618518 3890 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.618966 3882 tablet_bootstrap.cc:492] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: No bootstrap required, opened a new log
I20260925 19:09:59.619021 3882 ts_tablet_manager.cc:1397] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:59.619194 3882 raft_consensus.cc:348] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.619277 3882 raft_consensus.cc:374] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.619297 3882 raft_consensus.cc:729] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c54501e714bdba43d279254c173a3, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.619340 3882 consensus_queue.cc:260] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.619433 3882 ts_tablet_manager.cc:1428] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:59.619563 3887 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.619663 3887 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.682487 3887 raft_consensus.cc:482] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:59.682578 3887 raft_consensus.cc:504] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.682729 3887 leader_election.cc:288] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.682972 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071414cbd6d94f4b815a7562035120c8" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc" is_pre_election: true
I20260925 19:09:59.683058 3528 raft_consensus.cc:2454] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.682981 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071414cbd6d94f4b815a7562035120c8" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" is_pre_election: true
I20260925 19:09:59.683116 3668 raft_consensus.cc:2454] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.683282 3739 leader_election.cc:302] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.683403 3887 raft_consensus.cc:2781] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:59.683459 3887 raft_consensus.cc:482] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:59.683475 3887 raft_consensus.cc:3037] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.684448 3887 raft_consensus.cc:504] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.684547 3887 leader_election.cc:288] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Requested vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.684767 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071414cbd6d94f4b815a7562035120c8" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc"
I20260925 19:09:59.684840 3528 raft_consensus.cc:3037] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.684794 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071414cbd6d94f4b815a7562035120c8" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
I20260925 19:09:59.684893 3668 raft_consensus.cc:3037] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.685420 3528 raft_consensus.cc:2454] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.685590 3668 raft_consensus.cc:2454] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.685639 3739 leader_election.cc:302] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8297d071cc04dacac406c94e51c25cc, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.685729 3887 raft_consensus.cc:2781] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:59.685771 3887 raft_consensus.cc:686] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [term 1 LEADER]: Becoming Leader. State: Replica: e49c54501e714bdba43d279254c173a3, State: Running, Role: LEADER
I20260925 19:09:59.685819 3887 consensus_queue.cc:237] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.686170 3911 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: TxnStatusTablet state changed. Reason: New leader e49c54501e714bdba43d279254c173a3. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.686271 3911 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:59.686475 3923 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:09:59.686563 3923 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:09:59.686559 3370 catalog_manager.cc:5495] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49c54501e714bdba43d279254c173a3 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } health_report { overall_health: HEALTHY } } }
I20260925 19:09:59.695091 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.692694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.27.174.67:33403)
0925 19:09:59.692735 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.692737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.692746 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.692824 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.692825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.692841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.692902 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.693043 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.693526 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.694266 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.694484 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.694569 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.694762 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.694859 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.694936 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.694992 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.695014 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:59.695106 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.692611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:33403 (local address 127.0.0.1:60476)
0925 19:09:59.692682 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.692694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.692743 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.692911 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.692912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.692913 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:59.692959 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.692962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.693611 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.693612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.694182 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.694184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.694570 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.694571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.694643 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.694664 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.694923 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.694931 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.694987 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.695036 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:09:59.696144 3528 raft_consensus.cc:1261] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.696262 3668 raft_consensus.cc:1261] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.696367 3887 consensus_queue.cc:1035] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.696487 3911 consensus_queue.cc:1035] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.697033 3890 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: TxnStatusTablet state changed. Reason: New leader e49c54501e714bdba43d279254c173a3. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.697067 3888 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: TxnStatusTablet state changed. Reason: New leader e49c54501e714bdba43d279254c173a3. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.697146 3890 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.697178 3888 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.697691 3887 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.697775 3887 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:59.697867 3887 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.697925 3887 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P e49c54501e714bdba43d279254c173a3: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:09:59.698060 3895 mvcc.cc:205] Tried to move back new op lower bound from 7333328485153423360 to 7333328485113561088. Current Snapshot: MvccSnapshot[applied={T|T < 7333328485153423360}]
I20260925 19:09:59.698148 3888 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.698227 3888 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P 7259cb25fc2c4012bc43a5e7762aa5f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.698709 3896 mvcc.cc:205] Tried to move back new op lower bound from 7333328485153423360 to 7333328485113561088. Current Snapshot: MvccSnapshot[applied={T|T < 7333328485153423360}]
I20260925 19:09:59.698799 3890 tablet_replica.cc:439] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } } }
I20260925 19:09:59.698872 3890 tablet_replica.cc:442] T 071414cbd6d94f4b815a7562035120c8 P a8297d071cc04dacac406c94e51c25cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:09:59.702029 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.699663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.27.174.67:33403)
0925 19:09:59.699802 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.699804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.699813 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.699884 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.699885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.699896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.699944 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.700065 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.700546 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.701290 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.701531 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.701616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.701770 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.701809 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.701868 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.701929 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.701948 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 19:09:59.707228 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.704531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50879 (local address 127.27.174.67:33403)
0925 19:09:59.704567 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.704569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.704627 (+ 58us) server_negotiation.cc:407] Connection header received
0925 19:09:59.704673 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.704674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.704682 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.704751 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.704862 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.705577 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.706483 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.706777 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.706846 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.706982 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.707012 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.707072 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.707121 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.707138 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260925 19:09:59.707238 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.704468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:33403 (local address 127.27.174.67:50879)
0925 19:09:59.704574 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.704586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.704634 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.704753 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.704753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.704754 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:59.704803 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.704804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.705672 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.705672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.706410 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.706412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.706849 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.706850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.706897 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.706909 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.707078 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.707080 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.707120 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.707150 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 19:09:59.709482 3668 raft_consensus.cc:1261] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.709486 3528 raft_consensus.cc:1261] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.709714 3911 consensus_queue.cc:1035] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.709856 3930 consensus_queue.cc:1035] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.721302 3371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.721415 3371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.722087 3371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Sending DeleteTablet for 3 replicas of tablet d24866505db14ca2af9d82e6e025bb29
I20260925 19:09:59.722340 3508 tablet_service.cc:1514] Processing DeleteTablet for tablet d24866505db14ca2af9d82e6e025bb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35670
I20260925 19:09:59.722430 3906 tablet_replica.cc:330] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: stopping tablet replica
I20260925 19:09:59.722432 3648 tablet_service.cc:1514] Processing DeleteTablet for tablet d24866505db14ca2af9d82e6e025bb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46984
I20260925 19:09:59.722473 3906 raft_consensus.cc:2229] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.722505 3906 raft_consensus.cc:2258] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.722530 3905 tablet_replica.cc:330] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: stopping tablet replica
I20260925 19:09:59.722566 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet d24866505db14ca2af9d82e6e025bb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60472
I20260925 19:09:59.722584 3905 raft_consensus.cc:2229] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.722618 3905 raft_consensus.cc:2258] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.722694 3904 tablet_replica.cc:330] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: stopping tablet replica
I20260925 19:09:59.722745 3904 raft_consensus.cc:2229] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:59.722826 3904 raft_consensus.cc:2258] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.722873 3906 ts_tablet_manager.cc:1905] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.723093 3904 ts_tablet_manager.cc:1905] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.724646 3906 ts_tablet_manager.cc:1918] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:59.724648 3904 ts_tablet_manager.cc:1918] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:59.724709 3906 log.cc:1196] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/d24866505db14ca2af9d82e6e025bb29
I20260925 19:09:59.724720 3904 log.cc:1196] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/d24866505db14ca2af9d82e6e025bb29
I20260925 19:09:59.724937 3906 ts_tablet_manager.cc:1939] T d24866505db14ca2af9d82e6e025bb29 P a8297d071cc04dacac406c94e51c25cc: Deleting consensus metadata
I20260925 19:09:59.724941 3904 ts_tablet_manager.cc:1939] T d24866505db14ca2af9d82e6e025bb29 P e49c54501e714bdba43d279254c173a3: Deleting consensus metadata
I20260925 19:09:59.725238 3355 catalog_manager.cc:4841] TS a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049): tablet d24866505db14ca2af9d82e6e025bb29 (table test-table [id=af1f4ced7d9944498879da78d76153a2]) successfully deleted
I20260925 19:09:59.725320 3358 catalog_manager.cc:4841] TS e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403): tablet d24866505db14ca2af9d82e6e025bb29 (table test-table [id=af1f4ced7d9944498879da78d76153a2]) successfully deleted
I20260925 19:09:59.725546 3905 ts_tablet_manager.cc:1905] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.726953 3905 ts_tablet_manager.cc:1918] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:59.727002 3905 log.cc:1196] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/d24866505db14ca2af9d82e6e025bb29
I20260925 19:09:59.727212 3905 ts_tablet_manager.cc:1939] T d24866505db14ca2af9d82e6e025bb29 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting consensus metadata
I20260925 19:09:59.727517 3355 catalog_manager.cc:4841] TS 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339): tablet d24866505db14ca2af9d82e6e025bb29 (table test-table [id=af1f4ced7d9944498879da78d76153a2]) successfully deleted
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:09:59.746934 3433 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:59.741448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.27.174.126:33067)
0925 19:09:59.741499 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.741501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.741669 (+ 168us) server_negotiation.cc:407] Connection header received
0925 19:09:59.741734 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.741734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.741746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.741817 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.742395 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.742886 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.743661 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.743854 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.743940 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.746864 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44416: BlockingRecv error: recv got EOF from 127.0.0.1:44416 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 19:09:59 dist-test-slave-fcdn krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20260925 19:09:59.755142 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.747768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.27.174.126:33067)
0925 19:09:59.747809 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.747811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.747839 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:09:59.747916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.747917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.747928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.747992 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:59.748413 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.749168 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.749817 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.750141 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.750210 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.752316 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:59.752318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:59.752318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:59.752325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:59.753987 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.754357 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.754358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.754358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.754418 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:59.754621 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:59.754622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:59.754623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:59.754754 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:59.754848 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.754956 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.755042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:59.756959 3371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:59.757139 3371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:59.758563 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 (DEFAULT_TABLE table=test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.758698 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42193637ab344c5ab7b0e9cf75e59b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.760025 3882 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Bootstrap starting.
I20260925 19:09:59.760083 3648 tablet_service.cc:1467] Processing CreateTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 (DEFAULT_TABLE table=test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.760200 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42193637ab344c5ab7b0e9cf75e59b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.760442 3882 tablet_bootstrap.cc:654] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.761174 3882 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: No bootstrap required, opened a new log
I20260925 19:09:59.761262 3882 ts_tablet_manager.cc:1397] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.761387 3882 raft_consensus.cc:348] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.761477 3882 raft_consensus.cc:374] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.761538 3882 raft_consensus.cc:729] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c54501e714bdba43d279254c173a3, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.761521 3508 tablet_service.cc:1467] Processing CreateTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 (DEFAULT_TABLE table=test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.761577 3882 consensus_queue.cc:260] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.761633 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42193637ab344c5ab7b0e9cf75e59b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.761682 3882 ts_tablet_manager.cc:1428] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:59.762892 3883 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Bootstrap starting.
I20260925 19:09:59.762924 3886 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Bootstrap starting.
I20260925 19:09:59.763250 3883 tablet_bootstrap.cc:654] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.763338 3886 tablet_bootstrap.cc:654] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.763893 3883 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: No bootstrap required, opened a new log
I20260925 19:09:59.763948 3883 ts_tablet_manager.cc:1397] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:59.764043 3886 tablet_bootstrap.cc:492] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: No bootstrap required, opened a new log
I20260925 19:09:59.764066 3883 raft_consensus.cc:348] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.764118 3886 ts_tablet_manager.cc:1397] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.764139 3883 raft_consensus.cc:374] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.764164 3883 raft_consensus.cc:729] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.764204 3883 consensus_queue.cc:260] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.764299 3883 ts_tablet_manager.cc:1428] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:59.764273 3886 raft_consensus.cc:348] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.764338 3886 raft_consensus.cc:374] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.764364 3886 raft_consensus.cc:729] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8297d071cc04dacac406c94e51c25cc, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.764415 3886 consensus_queue.cc:260] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.764508 3886 ts_tablet_manager.cc:1428] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.868242 3933 raft_consensus.cc:482] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:59.868356 3933 raft_consensus.cc:504] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.868506 3933 leader_election.cc:288] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.868763 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42193637ab344c5ab7b0e9cf75e59b02" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc" is_pre_election: true
I20260925 19:09:59.868834 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42193637ab344c5ab7b0e9cf75e59b02" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" is_pre_election: true
I20260925 19:09:59.868892 3528 raft_consensus.cc:2454] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.868911 3668 raft_consensus.cc:2454] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 0.
I20260925 19:09:59.869057 3739 leader_election.cc:302] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8297d071cc04dacac406c94e51c25cc, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.869180 3933 raft_consensus.cc:2781] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:59.869222 3933 raft_consensus.cc:482] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:59.869244 3933 raft_consensus.cc:3037] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.869932 3933 raft_consensus.cc:504] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.870067 3933 leader_election.cc:288] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Requested vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339)
I20260925 19:09:59.870261 3668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42193637ab344c5ab7b0e9cf75e59b02" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5"
I20260925 19:09:59.870278 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42193637ab344c5ab7b0e9cf75e59b02" candidate_uuid: "e49c54501e714bdba43d279254c173a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc"
I20260925 19:09:59.870348 3528 raft_consensus.cc:3037] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.870357 3668 raft_consensus.cc:3037] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.870971 3668 raft_consensus.cc:2454] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.871109 3528 raft_consensus.cc:2454] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c54501e714bdba43d279254c173a3 in term 1.
I20260925 19:09:59.871147 3739 leader_election.cc:302] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.871255 3933 raft_consensus.cc:2781] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:59.871321 3933 raft_consensus.cc:686] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 1 LEADER]: Becoming Leader. State: Replica: e49c54501e714bdba43d279254c173a3, State: Running, Role: LEADER
I20260925 19:09:59.871372 3933 consensus_queue.cc:237] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.871987 3370 catalog_manager.cc:5495] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49c54501e714bdba43d279254c173a3 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "e49c54501e714bdba43d279254c173a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } health_report { overall_health: HEALTHY } } }
I20260925 19:09:59.889510 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.886154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.27.174.67:33403)
0925 19:09:59.886196 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.886198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.887083 (+ 885us) server_negotiation.cc:407] Connection header received
0925 19:09:59.887114 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.887115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.887127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.887172 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.887481 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.888016 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.888801 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.889005 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.889082 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.889238 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.889279 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.889340 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.889402 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.889420 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:59.890296 3528 raft_consensus.cc:1261] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.890264 3668 raft_consensus.cc:1261] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Refusing update from remote peer e49c54501e714bdba43d279254c173a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.890509 3933 consensus_queue.cc:1035] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.890626 3887 consensus_queue.cc:1035] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.894131 3370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.894232 3370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.894891 3370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Sending DeleteTablet for 3 replicas of tablet 42193637ab344c5ab7b0e9cf75e59b02
I20260925 19:09:59.895153 3648 tablet_service.cc:1514] Processing DeleteTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46984
I20260925 19:09:59.895162 3508 tablet_service.cc:1514] Processing DeleteTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35670
I20260925 19:09:59.895244 3905 tablet_replica.cc:330] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: stopping tablet replica
I20260925 19:09:59.895255 3906 tablet_replica.cc:330] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: stopping tablet replica
I20260925 19:09:59.895289 3905 raft_consensus.cc:2229] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.895304 3906 raft_consensus.cc:2229] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.895295 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 42193637ab344c5ab7b0e9cf75e59b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60472
I20260925 19:09:59.895327 3905 raft_consensus.cc:2258] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.895344 3906 raft_consensus.cc:2258] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.895370 3904 tablet_replica.cc:330] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: stopping tablet replica
I20260925 19:09:59.895413 3904 raft_consensus.cc:2229] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:59.895484 3904 raft_consensus.cc:2258] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.895743 3906 ts_tablet_manager.cc:1905] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.895938 3904 ts_tablet_manager.cc:1905] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.897217 3905 ts_tablet_manager.cc:1905] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.897205 3370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:09:59.897338 3370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:09:59.897538 3904 ts_tablet_manager.cc:1918] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.897589 3904 log.cc:1196] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/42193637ab344c5ab7b0e9cf75e59b02
I20260925 19:09:59.897790 3904 ts_tablet_manager.cc:1939] T 42193637ab344c5ab7b0e9cf75e59b02 P e49c54501e714bdba43d279254c173a3: Deleting consensus metadata
I20260925 19:09:59.898054 3358 catalog_manager.cc:4841] TS e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403): tablet 42193637ab344c5ab7b0e9cf75e59b02 (table test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]) successfully deleted
I20260925 19:09:59.898676 3648 tablet_service.cc:1467] Processing CreateTablet for tablet aa120573376141fa83975615e967cd60 (DEFAULT_TABLE table=test-table [id=6a0417de091d4ecaa3866468fb9e58fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.898751 3905 ts_tablet_manager.cc:1918] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.898762 3788 tablet_service.cc:1467] Processing CreateTablet for tablet aa120573376141fa83975615e967cd60 (DEFAULT_TABLE table=test-table [id=6a0417de091d4ecaa3866468fb9e58fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.898783 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa120573376141fa83975615e967cd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.898805 3905 log.cc:1196] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/42193637ab344c5ab7b0e9cf75e59b02
I20260925 19:09:59.898849 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa120573376141fa83975615e967cd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.899040 3905 ts_tablet_manager.cc:1939] T 42193637ab344c5ab7b0e9cf75e59b02 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting consensus metadata
I20260925 19:09:59.899374 3355 catalog_manager.cc:4841] TS 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339): tablet 42193637ab344c5ab7b0e9cf75e59b02 (table test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]) successfully deleted
I20260925 19:09:59.899935 3508 tablet_service.cc:1467] Processing CreateTablet for tablet aa120573376141fa83975615e967cd60 (DEFAULT_TABLE table=test-table [id=6a0417de091d4ecaa3866468fb9e58fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:09:59.900044 3883 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Bootstrap starting.
I20260925 19:09:59.900089 3882 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Bootstrap starting.
I20260925 19:09:59.900091 3906 ts_tablet_manager.cc:1918] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:09:59.900133 3906 log.cc:1196] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/42193637ab344c5ab7b0e9cf75e59b02
I20260925 19:09:59.900348 3906 ts_tablet_manager.cc:1939] T 42193637ab344c5ab7b0e9cf75e59b02 P a8297d071cc04dacac406c94e51c25cc: Deleting consensus metadata
I20260925 19:09:59.900355 3883 tablet_bootstrap.cc:654] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.900355 3882 tablet_bootstrap.cc:654] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.900468 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa120573376141fa83975615e967cd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:59.900869 3882 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: No bootstrap required, opened a new log
I20260925 19:09:59.900914 3882 ts_tablet_manager.cc:1397] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:59.901041 3882 raft_consensus.cc:348] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.901129 3882 raft_consensus.cc:374] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.901156 3882 raft_consensus.cc:729] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c54501e714bdba43d279254c173a3, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.901199 3882 consensus_queue.cc:260] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.901305 3882 ts_tablet_manager.cc:1428] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:59.901412 3883 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: No bootstrap required, opened a new log
I20260925 19:09:59.901448 3883 ts_tablet_manager.cc:1397] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:59.901556 3883 raft_consensus.cc:348] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.901641 3883 raft_consensus.cc:374] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.901661 3883 raft_consensus.cc:729] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.901659 3886 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: Bootstrap starting.
I20260925 19:09:59.901703 3883 consensus_queue.cc:260] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.901786 3883 ts_tablet_manager.cc:1428] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:59.901964 3355 catalog_manager.cc:4841] TS a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049): tablet 42193637ab344c5ab7b0e9cf75e59b02 (table test-table [id=a0baa46b587d464d9d6f916d5e06e9c8]) successfully deleted
I20260925 19:09:59.902249 3886 tablet_bootstrap.cc:654] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:59.903857 3886 tablet_bootstrap.cc:492] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: No bootstrap required, opened a new log
I20260925 19:09:59.903945 3886 ts_tablet_manager.cc:1397] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:09:59.904078 3886 raft_consensus.cc:348] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.904150 3886 raft_consensus.cc:374] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:59.904171 3886 raft_consensus.cc:729] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8297d071cc04dacac406c94e51c25cc, State: Initialized, Role: FOLLOWER
I20260925 19:09:59.904232 3886 consensus_queue.cc:260] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.904326 3886 ts_tablet_manager.cc:1428] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:59.952314 3888 raft_consensus.cc:482] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:09:59.952428 3888 raft_consensus.cc:504] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.952634 3888 leader_election.cc:288] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403)
I20260925 19:09:59.952822 3808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa120573376141fa83975615e967cd60" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c54501e714bdba43d279254c173a3" is_pre_election: true
I20260925 19:09:59.952845 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa120573376141fa83975615e967cd60" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc" is_pre_election: true
I20260925 19:09:59.952936 3808 raft_consensus.cc:2454] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 0.
I20260925 19:09:59.952935 3528 raft_consensus.cc:2454] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 0.
I20260925 19:09:59.953096 3602 leader_election.cc:302] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, e49c54501e714bdba43d279254c173a3; no voters:
I20260925 19:09:59.953228 3888 raft_consensus.cc:2781] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:09:59.953284 3888 raft_consensus.cc:482] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:09:59.953298 3888 raft_consensus.cc:3037] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.953986 3888 raft_consensus.cc:504] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.954123 3888 leader_election.cc:288] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 election: Requested vote from peers a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049), e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403)
I20260925 19:09:59.954253 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa120573376141fa83975615e967cd60" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8297d071cc04dacac406c94e51c25cc"
I20260925 19:09:59.954329 3528 raft_consensus.cc:3037] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.954267 3808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa120573376141fa83975615e967cd60" candidate_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c54501e714bdba43d279254c173a3"
I20260925 19:09:59.954358 3808 raft_consensus.cc:3037] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:59.954908 3528 raft_consensus.cc:2454] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 1.
I20260925 19:09:59.955078 3808 raft_consensus.cc:2454] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7259cb25fc2c4012bc43a5e7762aa5f5 in term 1.
I20260925 19:09:59.955081 3599 leader_election.cc:302] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7259cb25fc2c4012bc43a5e7762aa5f5, a8297d071cc04dacac406c94e51c25cc; no voters:
I20260925 19:09:59.955251 3888 raft_consensus.cc:2781] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:59.955368 3888 raft_consensus.cc:686] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 7259cb25fc2c4012bc43a5e7762aa5f5, State: Running, Role: LEADER
I20260925 19:09:59.955446 3888 consensus_queue.cc:237] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } }
I20260925 19:09:59.956000 3371 catalog_manager.cc:5495] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7259cb25fc2c4012bc43a5e7762aa5f5" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 } health_report { overall_health: UNKNOWN } } }
I20260925 19:09:59.965483 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.963245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.27.174.66:46339)
0925 19:09:59.963282 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.963284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.963291 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:59.963314 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.963314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.963324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.963378 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.963485 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.964023 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.964750 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.964953 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.965038 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.965192 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.965227 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.965314 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.965376 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.965396 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:09:59.970100 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.967925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38659 (local address 127.27.174.67:33403)
0925 19:09:59.967999 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.968001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.968154 (+ 153us) server_negotiation.cc:407] Connection header received
0925 19:09:59.968191 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.968191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.968200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.968238 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.968330 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.968785 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.969570 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.969742 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.969790 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.969922 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.969944 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.969981 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.970032 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.970046 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:09:59.970121 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.967848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:33403 (local address 127.27.174.66:38659)
0925 19:09:59.968077 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.968087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.968145 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.968240 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.968240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.968242 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:59.968279 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.968280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.968867 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.968868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.969448 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.969449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.969803 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.969804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.969843 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.969855 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.969994 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.969995 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.970030 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.970067 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260925 19:09:59.972355 3808 raft_consensus.cc:1261] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Refusing update from remote peer 7259cb25fc2c4012bc43a5e7762aa5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.972362 3528 raft_consensus.cc:1261] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Refusing update from remote peer 7259cb25fc2c4012bc43a5e7762aa5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:59.972656 3957 consensus_queue.cc:1035] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8297d071cc04dacac406c94e51c25cc" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.972788 3888 consensus_queue.cc:1035] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49c54501e714bdba43d279254c173a3" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:59.979894 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.977492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:46339 (local address 127.27.174.67:52563)
0925 19:09:59.977547 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.977559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:59.977595 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:59.977731 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:59.977732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:59.977734 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:09:59.977775 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.977777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.978391 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.978392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.979057 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:59.979059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.979425 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.979426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:59.979488 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.979493 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.979766 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.979768 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:09:59.979804 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 19:09:59.979839 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:09:59.979892 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.977554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52563 (local address 127.27.174.66:46339)
0925 19:09:59.977609 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:59.977611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:59.977620 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:59.977642 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:59.977642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:59.977653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:59.977727 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:59.977827 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.978311 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.979149 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:59.979353 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:59.979429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:59.979659 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:59.979697 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:59.979751 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:59.979825 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:09:59.979842 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:09:59.986105 3370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.986255 3370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:09:59.986867 3370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7141cc4084d4d0ebe689926053a4d6d: Sending DeleteTablet for 3 replicas of tablet aa120573376141fa83975615e967cd60
I20260925 19:09:59.987133 3508 tablet_service.cc:1514] Processing DeleteTablet for tablet aa120573376141fa83975615e967cd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35670
I20260925 19:09:59.987171 3648 tablet_service.cc:1514] Processing DeleteTablet for tablet aa120573376141fa83975615e967cd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46984
I20260925 19:09:59.987231 3906 tablet_replica.cc:330] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc: stopping tablet replica
I20260925 19:09:59.987272 3906 raft_consensus.cc:2229] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.987284 3905 tablet_replica.cc:330] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: stopping tablet replica
I20260925 19:09:59.987310 3906 raft_consensus.cc:2258] T aa120573376141fa83975615e967cd60 P a8297d071cc04dacac406c94e51c25cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.987352 3905 raft_consensus.cc:2229] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:09:59.987411 3905 raft_consensus.cc:2258] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.987426 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 3443
I20260925 19:09:59.987684 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet aa120573376141fa83975615e967cd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:09:59 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60472
I20260925 19:09:59.988080 3904 tablet_replica.cc:330] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: stopping tablet replica
I20260925 19:09:59.988118 3904 raft_consensus.cc:2229] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:09:59.988140 3904 raft_consensus.cc:2258] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:09:59.988451 3904 ts_tablet_manager.cc:1905] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.989877 3904 ts_tablet_manager.cc:1918] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:59.989952 3904 log.cc:1196] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/aa120573376141fa83975615e967cd60
I20260925 19:09:59.990165 3904 ts_tablet_manager.cc:1939] T aa120573376141fa83975615e967cd60 P e49c54501e714bdba43d279254c173a3: Deleting consensus metadata
I20260925 19:09:59.990492 3358 catalog_manager.cc:4841] TS e49c54501e714bdba43d279254c173a3 (127.27.174.67:33403): tablet aa120573376141fa83975615e967cd60 (table test-table [id=6a0417de091d4ecaa3866468fb9e58fb]) successfully deleted
I20260925 19:09:59.990689 3905 ts_tablet_manager.cc:1905] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:09:59.991838 3905 ts_tablet_manager.cc:1918] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:09:59.991888 3905 log.cc:1196] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/aa120573376141fa83975615e967cd60
I20260925 19:09:59.992103 3905 ts_tablet_manager.cc:1939] T aa120573376141fa83975615e967cd60 P 7259cb25fc2c4012bc43a5e7762aa5f5: Deleting consensus metadata
I20260925 19:09:59.992415 3355 catalog_manager.cc:4841] TS 7259cb25fc2c4012bc43a5e7762aa5f5 (127.27.174.66:46339): tablet aa120573376141fa83975615e967cd60 (table test-table [id=6a0417de091d4ecaa3866468fb9e58fb]) successfully deleted
W20260925 19:09:59.994383 3355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:35049 (error 108)
I20260925 19:09:59.994680 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 3583
I20260925 19:09:59.996522 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:59.994780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35049 (local address 127.0.0.1:35674)
0925 19:09:59.996417 (+ 1637us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:59.996462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1617,"thread_start_us":37,"threads_started":1}
W20260925 19:09:59.996662 3355 catalog_manager.cc:4584] TS a8297d071cc04dacac406c94e51c25cc (127.27.174.65:35049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa120573376141fa83975615e967cd60: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35049: connect: Connection refused (error 111)
I20260925 19:10:00.000922 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 3723
I20260925 19:10:00.006659 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 3339
2026-09-25T19:10:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2573 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:10:00 dist-test-slave-fcdn krb5kdc[3969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:10:00 dist-test-slave-fcdn krb5kdc[3969](info): set up 2 sockets
Sep 25 19:10:00 dist-test-slave-fcdn krb5kdc[3969](info): commencing operation
krb5kdc: starting...
W20260925 19:10:01.048043 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:10:01 dist-test-slave-fcdn krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:10:01.167388 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37677
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:01.250741 3982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:01.251029 3982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:01.251070 3982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:01.252261 3982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:01.252338 3982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:01.252362 3982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:01.252382 3982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:01.252413 3982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:01.254595 3982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:01.254873 3982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:01.255105 3982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:01.259878 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:01.259931 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:01.260262 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:01.260413 3982 server_base.cc:1034] running on GCE node
I20260925 19:10:01.260660 3982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:01.260936 3982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:01.262101 3982 hybrid_clock.cc:648] HybridClock initialized: now 1790363401262096 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T19:10:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1257 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:10:01 dist-test-slave-fcdn krb5kdc[3995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:10:01 dist-test-slave-fcdn krb5kdc[3995](info): set up 2 sockets
Sep 25 19:10:01 dist-test-slave-fcdn krb5kdc[3995](info): commencing operation
krb5kdc: starting...
W20260925 19:10:02.344735 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:10:02.462661 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41403
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:02.542801 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:02.543030 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:02.543054 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:02.544234 4008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:02.544301 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:02.544317 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:02.544329 4008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:02.544341 4008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:02.545790 4008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:02.546047 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:02.546269 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:02.551070 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:02.551115 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:02.551108 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:02.551187 4008 server_base.cc:1034] running on GCE node
I20260925 19:10:02.551491 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:02.551788 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:02.552979 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790363402552965 us; error 29 us; skew 500 ppm
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:02.555506 4008 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:10:02.555958 4008 webserver.cc:460] Webserver started at http://127.27.174.126:38189/ using document root <none> and password file <none>
I20260925 19:10:02.556187 4008 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:02.556255 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:02.556404 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:02.557502 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "07d0e56e95b046c287c52ab1f3924a80"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "b43f96ba9acf95e101f2560dd4c6e742"
server_key_iv: "fa2c07bf70d2f619ab961d522c3b3e74"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.557858 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "07d0e56e95b046c287c52ab1f3924a80"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "b43f96ba9acf95e101f2560dd4c6e742"
server_key_iv: "fa2c07bf70d2f619ab961d522c3b3e74"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.559177 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:02.560002 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.560175 4008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:02.560281 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "07d0e56e95b046c287c52ab1f3924a80"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "b43f96ba9acf95e101f2560dd4c6e742"
server_key_iv: "fa2c07bf70d2f619ab961d522c3b3e74"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.560353 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:02.568853 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:02.570595 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:02.570751 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:02.574427 4008 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41403
I20260925 19:10:02.574553 4075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41403 every 8 connection(s)
I20260925 19:10:02.574824 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:10:02.575722 4076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:02.577183 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4008
I20260925 19:10:02.577296 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:10:02.577462 28345 external_mini_cluster.cc:1428] Setting key 9e15bc90b0e5bfcb2bd87c27feeccd68
I20260925 19:10:02.578018 4076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Bootstrap starting.
I20260925 19:10:02.578617 4076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:02.578975 4076 log.cc:824] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:02.579746 4076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: No bootstrap required, opened a new log
I20260925 19:10:02.581849 4076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:02.581995 4076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:02.582039 4076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d0e56e95b046c287c52ab1f3924a80, State: Initialized, Role: FOLLOWER
I20260925 19:10:02.582163 4076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:02.582244 4076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:10:02.582278 4076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:10:02.582329 4076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:02.582993 4076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:02.583146 4076 leader_election.cc:302] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d0e56e95b046c287c52ab1f3924a80; no voters:
I20260925 19:10:02.583328 4076 leader_election.cc:288] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:10:02.583366 4081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:10:02.583549 4081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 LEADER]: Becoming Leader. State: Replica: 07d0e56e95b046c287c52ab1f3924a80, State: Running, Role: LEADER
I20260925 19:10:02.583630 4076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:10:02.583693 4081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:02.584832 4083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07d0e56e95b046c287c52ab1f3924a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } } }
I20260925 19:10:02.584956 4083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:02.585364 4082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d0e56e95b046c287c52ab1f3924a80. Latest consensus state: current_term: 1 leader_uuid: "07d0e56e95b046c287c52ab1f3924a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } } }
W20260925 19:10:02.585517 4095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 19:10:02.585546 4082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: This master's current role is: LEADER
W20260925 19:10:02.585573 4095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:10:02.585844 4096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:10:02.586043 4096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:10:02.587677 4096 catalog_manager.cc:1347] Generated new cluster ID: 800846810cb14167ab8df55358496040
I20260925 19:10:02.587730 4096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:10:02.588958 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:02.577977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.27.174.126:41403)
0925 19:10:02.578152 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:02.578159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:02.578184 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:10:02.578590 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:02.578596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:02.578638 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:02.578795 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:02.579553 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.580549 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.581673 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.582093 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.582189 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:02.585305 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:02.585310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:02.585311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:02.585323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:02.587304 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.587715 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.587718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.587718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.587838 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.588067 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.588068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.588069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.588364 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:02.588512 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:02.588627 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 19:10:02.588763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260925 19:10:02.608655 4096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:10:02.609383 4096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:10:02.620604 4096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Generated new TSK 0
I20260925 19:10:02.620792 4096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:10:02.654676 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:02.736101 4102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:02.736308 4102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:02.736332 4102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:02.737505 4102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:02.737550 4102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:02.737576 4102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:10:02.739104 4102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:02.739332 4102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:02.739558 4102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:02.744629 4107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:02.744665 4108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:02.744673 4110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:02.744889 4102 server_base.cc:1034] running on GCE node
I20260925 19:10:02.745105 4102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:02.745338 4102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:02.746524 4102 hybrid_clock.cc:648] HybridClock initialized: now 1790363402746509 us; error 44 us; skew 500 ppm
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:02.749188 4102 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:10:02.749643 4102 webserver.cc:460] Webserver started at http://127.27.174.65:45623/ using document root <none> and password file <none>
I20260925 19:10:02.749864 4102 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:02.749909 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:02.750051 4102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:02.751096 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "069db37412c74ebc8f0ed10dbb7b0df8"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "ee4eb6302646b5dc593e359a9e8d4a99"
server_key_iv: "74ebde8bed4a0c8b7357a1339a9738af"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.751461 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "069db37412c74ebc8f0ed10dbb7b0df8"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "ee4eb6302646b5dc593e359a9e8d4a99"
server_key_iv: "74ebde8bed4a0c8b7357a1339a9738af"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.752815 4102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:02.753571 4117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.753798 4102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:02.753894 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "069db37412c74ebc8f0ed10dbb7b0df8"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "ee4eb6302646b5dc593e359a9e8d4a99"
server_key_iv: "74ebde8bed4a0c8b7357a1339a9738af"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.753966 4102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:02.775820 4102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:02.777647 4102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:02.777806 4102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:02.778044 4102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:02.778365 4102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:02.778422 4102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.778462 4102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:02.778491 4102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.784442 4102 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:34117
I20260925 19:10:02.784548 4230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:34117 every 8 connection(s)
I20260925 19:10:02.784870 4102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:10:02.789541 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4102
I20260925 19:10:02.789651 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:10:02.789848 28345 external_mini_cluster.cc:1428] Setting key c4649c1a0c6c9ff673141fb0b4a760b3
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:02.794404 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:02.785658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:36819 (local address 127.27.174.126:41403)
0925 19:10:02.785748 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:02.785750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:02.785960 (+ 210us) server_negotiation.cc:407] Connection header received
0925 19:10:02.786409 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:02.786411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:02.786420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:02.786475 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:02.787336 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.787892 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.788823 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.789007 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.789069 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:02.791471 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:02.791473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:02.791473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:02.791480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:02.793055 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.793457 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.793458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.793458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.793514 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.793767 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.793767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.793768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.793880 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:02.793955 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:02.794215 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 19:10:02.794301 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 19:10:02.794601 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:02.785624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:36819)
0925 19:10:02.785826 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:02.785867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:02.786327 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:02.786524 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:02.786527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:02.787072 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:02.787257 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:02.787262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.787966 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.787967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:02.788752 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:02.788754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.789088 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.789088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:02.789184 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:02.789620 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:02.789659 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:02.791375 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:02.793144 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:02.793151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:02.793153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:02.793367 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:02.793585 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:02.793586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:02.793586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:02.793697 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:02.793946 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:02.793951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:02.794086 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:10:02.794196 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:10:02.794344 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260925 19:10:02.795089 4231 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41403
I20260925 19:10:02.795179 4231 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:02.795361 4231 heartbeater.cc:511] Master 127.27.174.126:41403 requested a full tablet report, sending...
I20260925 19:10:02.795943 4040 ts_manager.cc:194] Registered new tserver with Master: 069db37412c74ebc8f0ed10dbb7b0df8 (127.27.174.65:34117)
I20260925 19:10:02.796873 4040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:36819
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:10:02.814985 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:02.899094 4236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:02.899335 4236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:02.899379 4236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:02.900882 4236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:02.900935 4236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:02.900965 4236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:10:02.902469 4236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:02.902669 4236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:02.902855 4236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:02.908031 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:02.908164 4236 server_base.cc:1034] running on GCE node
W20260925 19:10:02.908021 4242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:02.908021 4241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:02.908535 4236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:02.908771 4236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:02.909924 4236 hybrid_clock.cc:648] HybridClock initialized: now 1790363402909912 us; error 43 us; skew 500 ppm
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:02.912415 4236 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:10:02.912901 4236 webserver.cc:460] Webserver started at http://127.27.174.66:39243/ using document root <none> and password file <none>
I20260925 19:10:02.913126 4236 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:02.913180 4236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:02.913343 4236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:02.914403 4236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "f8930460a8a249adbfd686349f538f23"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "88aec4e61bf86d8b621bc624a2cd8176"
server_key_iv: "7b47f452df5e29f282e40adbdec6c394"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.914762 4236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "f8930460a8a249adbfd686349f538f23"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "88aec4e61bf86d8b621bc624a2cd8176"
server_key_iv: "7b47f452df5e29f282e40adbdec6c394"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.916153 4236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:02.916908 4251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.917181 4236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:02.917289 4236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "f8930460a8a249adbfd686349f538f23"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "88aec4e61bf86d8b621bc624a2cd8176"
server_key_iv: "7b47f452df5e29f282e40adbdec6c394"
server_key_version: "encryptionkey@0"
I20260925 19:10:02.917359 4236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:02.925659 4236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:02.927552 4236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:02.927734 4236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:02.927956 4236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:02.928246 4236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:02.928299 4236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.928339 4236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:02.928367 4236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:02.934136 4236 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45927
I20260925 19:10:02.934180 4364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45927 every 8 connection(s)
I20260925 19:10:02.934485 4236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:10:02.939177 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4236
I20260925 19:10:02.939292 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:10:02.939488 28345 external_mini_cluster.cc:1428] Setting key a284eecc31d247a14831ec0e88e7ab5c
Sep 25 19:10:02 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:02.943787 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:02.935190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44463 (local address 127.27.174.126:41403)
0925 19:10:02.935252 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:02.935254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:02.935429 (+ 175us) server_negotiation.cc:407] Connection header received
0925 19:10:02.935977 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:02.935978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:02.935992 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:02.936050 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:02.936896 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.937373 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.938303 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.938474 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.938529 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:02.940860 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:02.940862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:02.940862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:02.940870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:02.942309 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.942744 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.942746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.942747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.942810 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:02.943072 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:02.943072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:02.943073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:02.943191 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:02.943267 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:02.943578 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 19:10:02.943698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:10:02.943966 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:02.935163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:44463)
0925 19:10:02.935305 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:02.935344 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:02.935891 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:02.936101 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:02.936104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:02.936645 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:02.936825 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:02.936830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.937480 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.937481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:02.938231 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:02.938233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:02.938542 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:02.938543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:02.938637 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:02.939082 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:02.939090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:02.940759 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:02.942420 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:02.942427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:02.942429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:02.942654 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:02.942876 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:02.942877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:02.942877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:02.942991 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:02.943258 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:02.943264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:02.943427 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 19:10:02.943556 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 19:10:02.943737 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260925 19:10:02.944818 4365 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41403
I20260925 19:10:02.944923 4365 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:02.945107 4365 heartbeater.cc:511] Master 127.27.174.126:41403 requested a full tablet report, sending...
I20260925 19:10:02.945541 4040 ts_manager.cc:194] Registered new tserver with Master: f8930460a8a249adbfd686349f538f23 (127.27.174.66:45927)
I20260925 19:10:02.945927 4040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:44463
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:10:02.966445 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:03.049403 4370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:03.049647 4370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:03.049683 4370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:03.050848 4370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:03.050926 4370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:03.050971 4370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:10:03.052433 4370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:03.052708 4370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:03.052948 4370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:03.057951 4375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.057932 4378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.058001 4376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.058183 4370 server_base.cc:1034] running on GCE node
I20260925 19:10:03.058437 4370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:03.058682 4370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:03.059859 4370 hybrid_clock.cc:648] HybridClock initialized: now 1790363403059834 us; error 28 us; skew 500 ppm
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:03.062536 4370 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:10:03.062965 4370 webserver.cc:460] Webserver started at http://127.27.174.67:35037/ using document root <none> and password file <none>
I20260925 19:10:03.063220 4370 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:03.063287 4370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:03.063436 4370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:03.064543 4370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "5bcc0bf929154e6f995751ec0c5d067b"
format_stamp: "Formatted at 2026-09-25 19:10:03 on dist-test-slave-fcdn"
server_key: "7ebf2c9f51ff91638521f3ecbd1b5c20"
server_key_iv: "8cf3b796d5dcc4bbd3ff28b33190fc9d"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.064901 4370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "5bcc0bf929154e6f995751ec0c5d067b"
format_stamp: "Formatted at 2026-09-25 19:10:03 on dist-test-slave-fcdn"
server_key: "7ebf2c9f51ff91638521f3ecbd1b5c20"
server_key_iv: "8cf3b796d5dcc4bbd3ff28b33190fc9d"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.066179 4370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:10:03.066910 4385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.067099 4370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:03.067195 4370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "5bcc0bf929154e6f995751ec0c5d067b"
format_stamp: "Formatted at 2026-09-25 19:10:03 on dist-test-slave-fcdn"
server_key: "7ebf2c9f51ff91638521f3ecbd1b5c20"
server_key_iv: "8cf3b796d5dcc4bbd3ff28b33190fc9d"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.067279 4370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:03.092039 4370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:03.094465 4370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:03.094628 4370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:03.094863 4370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:03.095186 4370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:03.095240 4370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.095283 4370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:03.095310 4370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.101190 4370 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37233
I20260925 19:10:03.101230 4498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37233 every 8 connection(s)
I20260925 19:10:03.101569 4370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:03.110561 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.102273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:45663 (local address 127.27.174.126:41403)
0925 19:10:03.102334 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.102336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.102545 (+ 209us) server_negotiation.cc:407] Connection header received
0925 19:10:03.103013 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.103014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.103023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.103083 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.104081 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.104723 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.105690 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.105869 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.105919 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.107941 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:03.107943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:03.107943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:03.107949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:03.109367 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:03.109720 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:03.109720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:03.109721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:03.109770 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:03.110048 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:03.110048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:03.110048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:03.110151 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:03.110205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:03.110433 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 19:10:03.110492 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:10:03.110746 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.102243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:45663)
0925 19:10:03.102388 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.102463 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.102916 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.103135 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.103138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.103775 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.103969 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.103974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.104808 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.104808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.105610 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.105611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.105939 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.105939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.106043 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.106479 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.106486 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.107862 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:03.109447 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:03.109452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:03.109454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:03.109655 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:03.109849 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:03.109850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:03.109850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:03.109967 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:03.110209 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:03.110213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:03.110324 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 19:10:03.110408 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:10:03.110535 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260925 19:10:03.111246 4499 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41403
I20260925 19:10:03.111353 4499 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:03.111625 4499 heartbeater.cc:511] Master 127.27.174.126:41403 requested a full tablet report, sending...
I20260925 19:10:03.111635 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4370
I20260925 19:10:03.111727 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:10:03.111902 28345 external_mini_cluster.cc:1428] Setting key 549506b57bd5bb49af0bd9c69731760a
I20260925 19:10:03.112094 4040 ts_manager.cc:194] Registered new tserver with Master: 5bcc0bf929154e6f995751ec0c5d067b (127.27.174.67:37233)
I20260925 19:10:03.112571 4040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:45663
I20260925 19:10:03.112764 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
WARNING: no policy specified for oryx/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
WARNING: no policy specified for oryx/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
I20260925 19:10:03.180147 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4102
I20260925 19:10:03.184607 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4236
I20260925 19:10:03.188575 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4370
I20260925 19:10:03.192991 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4008
I20260925 19:10:03.197732 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41403
--webserver_interface=127.27.174.126
--webserver_port=38189
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:03.278069 4508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:03.278270 4508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:03.278302 4508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:03.279505 4508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:03.279553 4508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:03.279568 4508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:03.279580 4508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:03.279592 4508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:03.281119 4508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=38189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:03.281364 4508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:03.281601 4508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:03.286307 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.286310 4513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.286396 4508 server_base.cc:1034] running on GCE node
W20260925 19:10:03.286491 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.286769 4508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:03.287055 4508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:03.288214 4508 hybrid_clock.cc:648] HybridClock initialized: now 1790363403288211 us; error 35 us; skew 500 ppm
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:03.290529 4508 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:10:03.290951 4508 webserver.cc:460] Webserver started at http://127.27.174.126:38189/ using document root <none> and password file <none>
I20260925 19:10:03.291181 4508 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:03.291258 4508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:03.292632 4508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:03.293268 4523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.293431 4508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:03.293524 4508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "07d0e56e95b046c287c52ab1f3924a80"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "b43f96ba9acf95e101f2560dd4c6e742"
server_key_iv: "fa2c07bf70d2f619ab961d522c3b3e74"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.293844 4508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:03.307912 4508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:03.309674 4508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:03.309835 4508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:03.313624 4508 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41403
I20260925 19:10:03.313669 4575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41403 every 8 connection(s)
I20260925 19:10:03.314045 4508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:10:03.315419 4576 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:10:03.316118 4576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Bootstrap starting.
I20260925 19:10:03.322248 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4508
I20260925 19:10:03.322576 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:34117
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:41403
--webserver_port=45623
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:10:03.324501 4576 log.cc:824] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:03.326185 4576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:10:03.326489 4576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Bootstrap complete.
I20260925 19:10:03.328733 4576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:03.328969 4576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d0e56e95b046c287c52ab1f3924a80, State: Initialized, Role: FOLLOWER
I20260925 19:10:03.329133 4576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:03.329200 4576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:10:03.329260 4576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:10:03.329308 4576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:10:03.330302 4576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:03.330385 4576 leader_election.cc:302] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d0e56e95b046c287c52ab1f3924a80; no voters:
I20260925 19:10:03.330588 4576 leader_election.cc:288] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:10:03.330621 4580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:10:03.330730 4580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [term 2 LEADER]: Becoming Leader. State: Replica: 07d0e56e95b046c287c52ab1f3924a80, State: Running, Role: LEADER
I20260925 19:10:03.330854 4576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:10:03.330879 4580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } }
I20260925 19:10:03.331240 4581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07d0e56e95b046c287c52ab1f3924a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } } }
I20260925 19:10:03.331251 4582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d0e56e95b046c287c52ab1f3924a80. Latest consensus state: current_term: 2 leader_uuid: "07d0e56e95b046c287c52ab1f3924a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d0e56e95b046c287c52ab1f3924a80" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41403 } } }
I20260925 19:10:03.331343 4581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:03.331354 4582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:03.331560 4587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:10:03.332142 4587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:10:03.332291 4587 catalog_manager.cc:1259] Loaded cluster ID: 800846810cb14167ab8df55358496040
I20260925 19:10:03.332535 4587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:10:03.333727 4587 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:10:03.333932 4587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07d0e56e95b046c287c52ab1f3924a80: Loaded TSK: 0
I20260925 19:10:03.334187 4587 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:10:03.403059 4578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:03.403261 4578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:03.403285 4578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:03.404477 4578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:03.404520 4578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:03.404551 4578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:10:03.405951 4578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:34117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=45623
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:03.406209 4578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:03.406445 4578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:03.411383 4605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.411387 4602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.411371 4603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.411670 4578 server_base.cc:1034] running on GCE node
I20260925 19:10:03.411878 4578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:03.412163 4578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:03.413355 4578 hybrid_clock.cc:648] HybridClock initialized: now 1790363403413339 us; error 43 us; skew 500 ppm
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:03.416320 4578 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20260925 19:10:03.416822 4578 webserver.cc:460] Webserver started at http://127.27.174.65:45623/ using document root <none> and password file <none>
I20260925 19:10:03.417074 4578 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:03.417132 4578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:03.418502 4578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:03.419130 4612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.419333 4578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:03.419443 4578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "069db37412c74ebc8f0ed10dbb7b0df8"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "ee4eb6302646b5dc593e359a9e8d4a99"
server_key_iv: "74ebde8bed4a0c8b7357a1339a9738af"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.419871 4578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:03.451965 4578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:03.454356 4578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:03.454540 4578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:03.454782 4578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:03.455081 4578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:03.455137 4578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.455176 4578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:03.455204 4578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.460827 4578 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:34117
I20260925 19:10:03.460875 4725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:34117 every 8 connection(s)
I20260925 19:10:03.461247 4578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.468080 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4578
I20260925 19:10:03.468374 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:45927
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:41403
--webserver_port=39243
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.469877 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.461950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:34379)
0925 19:10:03.462111 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.462151 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.462614 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.462987 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.462990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.463538 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.463747 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.463752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.464850 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.464851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.465622 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.465624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.466215 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.466216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.466327 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.466815 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.466823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.469634 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260925 19:10:03.470008 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.462050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34379 (local address 127.27.174.126:41403)
0925 19:10:03.462217 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.462233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.462262 (+ 29us) server_negotiation.cc:407] Connection header received
0925 19:10:03.462771 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.462776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.462816 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.462929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.463860 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.464752 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.465711 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.466107 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.466189 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.469808 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:34379: BlockingRecv error: recv got EOF from 127.27.174.65:34379 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260925 19:10:03.470098 4726 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41403 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41403: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.475088 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.470296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:39513)
0925 19:10:03.470335 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.470345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.470383 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.470537 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.470538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.470667 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.470733 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.470735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.471330 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.471331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.472257 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.472260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.472604 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.472605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.472679 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.473075 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.473081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.474998 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 19:10:03.475142 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.470325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39513 (local address 127.27.174.126:41403)
0925 19:10:03.470373 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.470375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.470385 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:03.470445 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.470446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.470457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.470535 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.470799 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.471252 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.472340 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.472523 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.472583 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.475076 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:39513: BlockingRecv error: recv got EOF from 127.27.174.65:39513 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.479936 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.475372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:44333)
0925 19:10:03.475416 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.475426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.475473 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.475676 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.475677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.475830 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.475866 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.475868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.476469 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.476471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.477345 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.477348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.477674 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.477675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.477739 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.478128 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.478134 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.479888 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:10:03.480083 4726 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:10:03.480211 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.475419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:44333 (local address 127.27.174.126:41403)
0925 19:10:03.475458 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.475461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.475468 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:10:03.475545 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.475545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.475553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.475636 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.475940 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.476380 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.477432 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.477602 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.477653 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.479991 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:44333: BlockingRecv error: recv got EOF from 127.27.174.65:44333 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 19:10:03.551467 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:03.551697 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:03.551720 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:03.552845 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:03.552884 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:03.552910 4730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:10:03.554255 4730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:45927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=39243
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:03.554513 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:03.554740 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:03.559926 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.559876 4735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.560030 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.560202 4730 server_base.cc:1034] running on GCE node
I20260925 19:10:03.560436 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:03.560708 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:03.561880 4730 hybrid_clock.cc:648] HybridClock initialized: now 1790363403561874 us; error 42 us; skew 500 ppm
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:03.564347 4730 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20260925 19:10:03.564814 4730 webserver.cc:460] Webserver started at http://127.27.174.66:39243/ using document root <none> and password file <none>
I20260925 19:10:03.565083 4730 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:03.565164 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:03.566505 4730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:03.567071 4745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.567252 4730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:10:03.567351 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "f8930460a8a249adbfd686349f538f23"
format_stamp: "Formatted at 2026-09-25 19:10:02 on dist-test-slave-fcdn"
server_key: "88aec4e61bf86d8b621bc624a2cd8176"
server_key_iv: "7b47f452df5e29f282e40adbdec6c394"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.567760 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:03.584802 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:03.586683 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:03.586846 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:03.587077 4730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:03.587383 4730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:03.587443 4730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.587477 4730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:03.587508 4730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.593137 4730 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45927
I20260925 19:10:03.593180 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45927 every 8 connection(s)
I20260925 19:10:03.593550 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:10:03.593791 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4730
I20260925 19:10:03.594103 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:37233
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:41403
--webserver_port=35037
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:10:03.601188 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.594310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37627 (local address 127.27.174.126:41403)
0925 19:10:03.594357 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.594359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.594600 (+ 241us) server_negotiation.cc:407] Connection header received
0925 19:10:03.595120 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.595121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.595138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.595192 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.596196 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.596761 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.597739 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.597946 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.598029 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.601141 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:37627: BlockingRecv error: recv got EOF from 127.27.174.66:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:10:03.601349 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.594303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:37627)
0925 19:10:03.594477 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.594511 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.595013 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.595265 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.595269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.595892 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.596087 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.596091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.596860 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.596861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.597647 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.597649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.598033 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.598034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.598151 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.598665 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.598673 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.601064 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
W20260925 19:10:03.601583 4859 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41403 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41403: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.606567 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.601821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:34817)
0925 19:10:03.601928 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.601941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.602002 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.602126 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.602126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.602290 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.602354 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.602356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.602947 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.602948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.603897 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.603899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.604245 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.604247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.604321 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.604765 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.604770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.606500 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 19:10:03.606660 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.601876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:34817 (local address 127.27.174.126:41403)
0925 19:10:03.601934 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.601935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.601984 (+ 49us) server_negotiation.cc:407] Connection header received
0925 19:10:03.602065 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.602066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.602073 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.602117 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.602436 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.602873 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.603988 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.604163 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.604231 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.606580 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:34817: BlockingRecv error: recv got EOF from 127.27.174.66:34817 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.611763 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.606997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:52367)
0925 19:10:03.607037 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.607047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.607113 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.607285 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.607286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.607423 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.607483 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.607484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.608442 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.608443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.609236 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.609238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.609568 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.609569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.609620 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.609981 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.609986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.611696 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:10:03.611841 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.607125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:52367 (local address 127.27.174.126:41403)
0925 19:10:03.607180 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.607182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.607188 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:10:03.607205 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.607206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.607213 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.607264 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.607571 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.608341 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.609312 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.609488 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.609551 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.611779 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:52367: BlockingRecv error: recv got EOF from 127.27.174.66:52367 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 19:10:03.611949 4859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:10:03.679263 4862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:03.679471 4862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:03.679494 4862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:03.680691 4862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:03.680742 4862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:03.680768 4862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:10:03.682142 4862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:37233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=35037
--tserver_master_addrs=127.27.174.126:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:03.682331 4862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:03.682503 4862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:10:03.687873 4862 server_base.cc:1034] running on GCE node
W20260925 19:10:03.687846 4870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.687839 4868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:03.687846 4867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:03.688313 4862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:03.688604 4862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:03.689769 4862 hybrid_clock.cc:648] HybridClock initialized: now 1790363403689760 us; error 28 us; skew 500 ppm
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363403, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:03.692389 4862 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20260925 19:10:03.692874 4862 webserver.cc:460] Webserver started at http://127.27.174.67:35037/ using document root <none> and password file <none>
I20260925 19:10:03.693112 4862 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:03.693176 4862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:03.694574 4862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:03.695159 4877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.695351 4862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:10:03.695441 4862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "5bcc0bf929154e6f995751ec0c5d067b"
format_stamp: "Formatted at 2026-09-25 19:10:03 on dist-test-slave-fcdn"
server_key: "7ebf2c9f51ff91638521f3ecbd1b5c20"
server_key_iv: "8cf3b796d5dcc4bbd3ff28b33190fc9d"
server_key_version: "encryptionkey@0"
I20260925 19:10:03.695902 4862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:03.722913 4862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:03.725099 4862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:03.725260 4862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:03.725522 4862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:03.725840 4862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:03.725895 4862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.725936 4862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:03.725965 4862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:03.732219 4862 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37233
I20260925 19:10:03.732259 4990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37233 every 8 connection(s)
I20260925 19:10:03.732606 4862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.739455 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 4862
W20260925 19:10:03.739888 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.733351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:55621 (local address 127.27.174.126:41403)
0925 19:10:03.733419 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.733421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.733630 (+ 209us) server_negotiation.cc:407] Connection header received
0925 19:10:03.734073 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.734074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.734082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.734140 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.735035 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.735507 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.736550 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.736736 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.736794 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.739819 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:55621: BlockingRecv error: recv got EOF from 127.27.174.67:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:10:03.739948 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.733327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:55621)
0925 19:10:03.733495 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.733537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.733992 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.734190 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.734194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.734756 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.734948 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.734953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.735587 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.735587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.736431 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.736433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.736809 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.736809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.736911 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.737362 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.737370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.739730 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260925 19:10:03.740273 4991 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41403 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41403: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.744951 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.740532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:34143)
0925 19:10:03.740577 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.740585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.740612 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.740813 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.740814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.740948 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.740988 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.740989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.741628 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.741629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.742129 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.742131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.742508 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.742509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.742574 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.743125 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.743129 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.744907 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:10:03.745087 4995 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.740581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34143 (local address 127.27.174.126:41403)
0925 19:10:03.740707 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.740708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.740714 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:10:03.740729 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.740730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.740737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.740806 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.741074 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.741537 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.742223 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.742426 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.742507 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.745017 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:34143: BlockingRecv error: recv got EOF from 127.27.174.67:34143 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260925 19:10:03.747413 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.740053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.27.174.126:41403)
0925 19:10:03.740112 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.740114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.740123 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:10:03.740173 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.740174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.740183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.740245 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.740920 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.741375 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.742144 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.742330 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.742413 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.743710 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:03.743714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:03.743715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:03.743724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:03.745877 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:03.746269 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:03.746273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:03.746274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:03.746414 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:03.746657 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:03.746658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:03.746658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:03.746944 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:03.747087 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:03.747160 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:10:03.747301 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:03 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:03.749831 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:03.745241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:37273)
0925 19:10:03.745292 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:03.745303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:03.745337 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:03.745468 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:03.745468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:03.745601 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:03.745634 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.745636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.746381 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.746382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.747160 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:03.747162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.747542 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.747543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:03.747641 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.748065 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:03.748071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:03.749788 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:10:03.749938 4995 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:03.745305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37273 (local address 127.27.174.126:41403)
0925 19:10:03.745355 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:03.745357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:03.745372 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:10:03.745389 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:03.745389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:03.745397 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:03.745443 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:03.745708 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.746302 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.747240 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:03.747444 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:03.747540 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:03.749871 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:37273: BlockingRecv error: recv got EOF from 127.27.174.67:37273 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 19:10:03.749976 4991 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:04.485776 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:04.480695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:51865)
0925 19:10:04.480873 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:04.480889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:04.480974 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:04.481130 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:04.481131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:04.481289 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:04.481337 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.481338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.482040 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.482041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.482573 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.482574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.482901 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.482902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.482939 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.483551 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:04.483558 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:04.485673 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
W20260925 19:10:04.485821 4997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:04.480826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:51865 (local address 127.27.174.126:41403)
0925 19:10:04.480970 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:04.480973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:04.480983 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:04.481034 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:04.481035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:04.481044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:04.481107 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:04.481440 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.481964 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.482646 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.482833 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.482896 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.485747 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:51865: BlockingRecv error: recv got EOF from 127.27.174.65:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:04.617229 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:04.612326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:47491)
0925 19:10:04.612463 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:04.612474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:04.612536 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:04.612691 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:04.612692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:04.612854 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:04.612900 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.612902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.613560 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.613561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.614106 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.614108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.614701 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.614702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.614758 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.615282 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:04.615292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:04.617165 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260925 19:10:04.617301 4997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:04.612436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47491 (local address 127.27.174.126:41403)
0925 19:10:04.612488 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:04.612490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:04.612522 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:10:04.612603 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:04.612604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:04.612614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:04.612672 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:04.612994 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.613485 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.614180 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.614527 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.614593 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.617254 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:47491: BlockingRecv error: recv got EOF from 127.27.174.66:47491 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:04 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:04.754875 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:04.750387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:59053)
0925 19:10:04.750536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:04.750547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:04.750609 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:04.750734 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:04.750735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:04.750890 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:04.750937 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.750938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.751576 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.751576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.752124 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:04.752125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.752496 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.752497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:04.752533 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.752986 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:04.752992 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:04.754794 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
W20260925 19:10:04.754940 4997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:04.750450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:59053 (local address 127.27.174.126:41403)
0925 19:10:04.750499 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:04.750501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:04.750614 (+ 113us) server_negotiation.cc:407] Connection header received
0925 19:10:04.750656 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:04.750657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:04.750668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:04.750707 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:04.751003 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.751508 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.752217 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:04.752419 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:04.752486 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:04.754890 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:59053: BlockingRecv error: recv got EOF from 127.27.174.67:59053 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:05.491132 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:05.486305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:55997)
0925 19:10:05.486451 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:05.486463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:05.486528 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:05.486685 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:05.486686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:05.486841 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:05.486889 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.486891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.487533 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.487534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.488143 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.488146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.488495 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.488496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.488537 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.489090 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:05.489095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:05.491037 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
W20260925 19:10:05.491185 5001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:05.486366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:55997 (local address 127.27.174.126:41403)
0925 19:10:05.486499 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:05.486501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:05.486511 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:05.486589 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:05.486590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:05.486600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:05.486663 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:05.486953 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.487445 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.488227 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.488427 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.488482 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.491113 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:55997: BlockingRecv error: recv got EOF from 127.27.174.65:55997 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:05.622773 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:05.617763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:44921)
0925 19:10:05.618049 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:05.618061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:05.618149 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:05.618346 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:05.618347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:05.618543 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:05.618591 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.618592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.619385 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.619386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.620125 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.620127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.620443 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.620444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.620482 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.620899 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:05.620903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:05.622680 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":103,"threads_started":1}
W20260925 19:10:05.622826 5001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:05.617866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44921 (local address 127.27.174.126:41403)
0925 19:10:05.618042 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:05.618044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:05.618130 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:10:05.618248 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:05.618249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:05.618260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:05.618326 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:05.618672 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.619272 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.620192 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.620376 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.620438 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.622764 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:44921: BlockingRecv error: recv got EOF from 127.27.174.66:44921 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:05 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:05.760174 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:05.755448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:34497)
0925 19:10:05.755640 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:05.755657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:05.755719 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:05.755866 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:05.755867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:05.756046 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:05.756092 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.756094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.756802 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.756803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.757338 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:05.757339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.757715 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.757716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:05.757784 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.758252 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:05.758256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:05.760065 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
W20260925 19:10:05.760212 5001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:05.755645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34497 (local address 127.27.174.126:41403)
0925 19:10:05.755697 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:05.755699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:05.755710 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:10:05.755778 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:05.755779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:05.755793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:05.755848 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:05.756189 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.756707 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.757426 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:05.757627 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:05.757714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:05.760148 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:34497: BlockingRecv error: recv got EOF from 127.27.174.67:34497 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:06.496004 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:06.491566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:38051)
0925 19:10:06.491726 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:06.491738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:06.491807 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:06.491920 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:06.491921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:06.492080 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:06.492125 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.492126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.492773 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.492774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.493299 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.493301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.493609 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.493610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.493649 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.494149 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:06.494154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:06.495926 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
W20260925 19:10:06.496130 5005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:06.491650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:38051 (local address 127.27.174.126:41403)
0925 19:10:06.491768 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:06.491770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:06.491791 (+ 21us) server_negotiation.cc:407] Connection header received
0925 19:10:06.491854 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:06.491855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:06.491863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:06.491916 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:06.492181 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.492691 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.493369 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.493549 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.493595 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.496032 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:38051: BlockingRecv error: recv got EOF from 127.27.174.65:38051 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:06.628191 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:06.623234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:50635)
0925 19:10:06.623388 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:06.623399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:06.623500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:06.623672 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:06.623673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:06.623840 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:06.623885 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.623887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.624618 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.624619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.625156 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.625158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.625534 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.625535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.625600 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.626069 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:06.626074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:06.628126 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20260925 19:10:06.628283 5005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:06.623299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:50635 (local address 127.27.174.126:41403)
0925 19:10:06.623376 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:06.623378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:06.623498 (+ 120us) server_negotiation.cc:407] Connection header received
0925 19:10:06.623567 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:06.623568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:06.623579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:06.623667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:06.623982 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.624514 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.625243 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.625448 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.625532 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.628218 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:50635: BlockingRecv error: recv got EOF from 127.27.174.66:50635 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:06 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:06.765074 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:06.760724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:56989)
0925 19:10:06.760867 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:06.760883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:06.760930 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:06.761068 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:06.761068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:06.761218 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:06.761263 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.761265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.761937 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.761938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.762454 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:06.762456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.762764 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.762765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:06.762815 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.763270 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:06.763274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:06.765016 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260925 19:10:06.765203 5005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:06.760860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56989 (local address 127.27.174.126:41403)
0925 19:10:06.760919 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:06.760926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:06.760935 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:10:06.760976 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:06.760976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:06.760987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:06.761041 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:06.761361 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.761859 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.762522 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:06.762702 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:06.762753 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:06.765120 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:56989: BlockingRecv error: recv got EOF from 127.27.174.67:56989 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:07.501559 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:07.496546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:53457)
0925 19:10:07.496686 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:07.496706 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:07.496761 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:07.496902 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:07.496903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:07.497143 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:07.497192 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.497194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.497879 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.497880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.498551 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.498553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.498940 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.498941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.499011 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.499642 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:07.499647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:07.501498 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
W20260925 19:10:07.501701 5009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:07.496596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:53457 (local address 127.27.174.126:41403)
0925 19:10:07.496766 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:07.496768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:07.496778 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:07.496802 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:07.496802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:07.496812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:07.496876 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:07.497265 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.497774 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.498627 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.498819 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.498932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.501588 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:53457: BlockingRecv error: recv got EOF from 127.27.174.65:53457 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:07.633380 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:07.628664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:57637)
0925 19:10:07.628838 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:07.628851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:07.628921 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:07.629102 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:07.629104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:07.629269 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:07.629313 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.629315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.629965 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.629967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.630489 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.630491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.630906 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.630907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.630947 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.631401 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:07.631405 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:07.633286 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
W20260925 19:10:07.633426 5009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:07.628802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57637 (local address 127.27.174.126:41403)
0925 19:10:07.628863 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:07.628865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:07.628975 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:10:07.629001 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:07.629002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:07.629013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:07.629082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:07.629388 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.629881 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.630631 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.630844 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.630899 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.633362 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:57637: BlockingRecv error: recv got EOF from 127.27.174.66:57637 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:07 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:07.770954 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:07.765540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:47951)
0925 19:10:07.765682 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:07.765694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:07.765758 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:07.765925 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:07.765926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:07.766078 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:07.766125 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.766126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.767110 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.767111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.767710 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:07.767712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.768203 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.768204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:07.768244 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.768780 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:07.768786 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:07.770852 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
W20260925 19:10:07.771068 5009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:07.765626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:47951 (local address 127.27.174.126:41403)
0925 19:10:07.765689 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:07.765691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:07.765762 (+ 71us) server_negotiation.cc:407] Connection header received
0925 19:10:07.765801 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:07.765801 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:07.765815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:07.765903 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:07.766189 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.767017 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.767782 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:07.768122 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:07.768202 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:07.770988 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:47951: BlockingRecv error: recv got EOF from 127.27.174.67:47951 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:08.506507 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:08.502050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:37663)
0925 19:10:08.502233 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:08.502244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:08.502309 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:08.502457 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:08.502458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:08.502627 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:08.502683 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.502685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.503309 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.503310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.503840 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.503842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.504181 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.504182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.504249 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.504750 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:08.504754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:08.506451 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
W20260925 19:10:08.506654 5013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:08.502108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:37663 (local address 127.27.174.126:41403)
0925 19:10:08.502264 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:08.502266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:08.502292 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:10:08.502363 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:08.502364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:08.502373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:08.502434 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:08.502746 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.503243 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.503907 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.504101 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.504155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.506560 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:37663: BlockingRecv error: recv got EOF from 127.27.174.65:37663 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:08.638521 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:08.633903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:46615)
0925 19:10:08.634057 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:08.634070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:08.634138 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:08.634329 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:08.634330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:08.634509 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:08.634554 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.634556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.635307 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.635308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.635856 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.635858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.636245 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.636246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.636310 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.636723 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:08.636727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:08.638464 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
W20260925 19:10:08.638625 5013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:08.634043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:46615 (local address 127.27.174.126:41403)
0925 19:10:08.634092 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:08.634094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:08.634120 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:10:08.634227 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:08.634228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:08.634239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:08.634322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:08.634635 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.635220 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.635943 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.636155 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.636242 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.638555 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:46615: BlockingRecv error: recv got EOF from 127.27.174.66:46615 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:08 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:08.781277 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:08.771625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:57563)
0925 19:10:08.771764 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:08.771776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:08.771839 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:08.771958 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:08.771958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:08.772113 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:08.772160 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.772161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.772793 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.772794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.773330 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:08.773332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.773677 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.773678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:08.773732 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.779360 (+ 5628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:08.779364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:08.781185 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260925 19:10:08.781308 5013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:08.771702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:57563 (local address 127.27.174.126:41403)
0925 19:10:08.771773 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:08.771775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:08.771828 (+ 53us) server_negotiation.cc:407] Connection header received
0925 19:10:08.771884 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:08.771885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:08.771896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:08.771950 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:08.772219 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.772712 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.773407 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:08.773591 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:08.773643 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:08.781249 (+ 7606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:57563: BlockingRecv error: recv got EOF from 127.27.174.67:57563 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:09.511763 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:09.507006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:40211)
0925 19:10:09.507156 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:09.507170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:09.507238 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:09.507373 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:09.507374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:09.507550 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:09.507625 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.507627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.508290 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.508291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.508930 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.508933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.509400 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.509401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.509465 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.509946 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:09.509951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:09.511706 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
W20260925 19:10:09.511909 5017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:09.507067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:40211 (local address 127.27.174.126:41403)
0925 19:10:09.507225 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:09.507227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:09.507236 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:10:09.507283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:09.507283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:09.507295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:09.507355 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:09.507700 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.508218 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.509004 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.509325 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.509389 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.511813 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:40211: BlockingRecv error: recv got EOF from 127.27.174.65:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:09.643334 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:09.639003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:59195)
0925 19:10:09.639124 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:09.639133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:09.639185 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:09.639287 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:09.639288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:09.639455 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:09.639512 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.639514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.640138 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.640139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.640660 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.640661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.640965 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.640966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.641013 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.641495 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:09.641500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:09.643239 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260925 19:10:09.643390 5017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:09.639034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:59195 (local address 127.27.174.126:41403)
0925 19:10:09.639080 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:09.639082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:09.639187 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:10:09.639225 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:09.639226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:09.639236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:09.639283 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:09.639568 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.640073 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.640728 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.640906 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.640964 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.643327 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:59195: BlockingRecv error: recv got EOF from 127.27.174.66:59195 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:09 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:09.787247 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:09.781803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:48367)
0925 19:10:09.781916 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:09.781962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:09.782049 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:09.782210 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:09.782211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:09.782433 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:09.782506 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.782508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.783309 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.783310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.783885 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:09.783887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.784418 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.784419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:09.784481 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.784911 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:09.784916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:09.787143 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260925 19:10:09.787426 5017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:09.781874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:48367 (local address 127.27.174.126:41403)
0925 19:10:09.781920 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:09.781923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:09.782047 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:10:09.782108 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:09.782109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:09.782127 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:09.782200 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:09.782573 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.783219 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.783979 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:09.784326 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:09.784416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:09.787277 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:48367: BlockingRecv error: recv got EOF from 127.27.174.67:48367 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:10.517136 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:10.512275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:37499)
0925 19:10:10.512420 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:10.512437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:10.512504 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:10.512690 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:10.512691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:10.512888 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:10.512949 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.512952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.513677 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.513678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.514215 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.514216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.514599 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.514600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.514665 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.515108 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:10.515113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:10.517034 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
W20260925 19:10:10.517201 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:10.512437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:37499 (local address 127.27.174.126:41403)
0925 19:10:10.512554 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:10.512555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:10.512563 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:10:10.512585 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:10.512585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:10.512595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:10.512652 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:10.513039 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.513589 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.514310 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.514517 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.514588 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.517126 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:37499: BlockingRecv error: recv got EOF from 127.27.174.65:37499 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:10.648137 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:10.643834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:49061)
0925 19:10:10.643973 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:10.643984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:10.644046 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:10.644163 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:10.644164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:10.644315 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:10.644362 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.644363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.644970 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.644971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.645487 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.645488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.645806 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.645807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.645844 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.646284 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:10.646288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:10.648082 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260925 19:10:10.648270 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:10.643889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:49061 (local address 127.27.174.126:41403)
0925 19:10:10.643941 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:10.643943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:10.644031 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:10:10.644092 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:10.644093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:10.644103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:10.644149 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:10.644422 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.644908 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.645558 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.645744 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.645798 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.648188 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:49061: BlockingRecv error: recv got EOF from 127.27.174.66:49061 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:10 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:10.792259 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:10.787727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:50467)
0925 19:10:10.787883 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:10.787895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:10.787962 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:10.788119 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:10.788120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:10.788296 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:10.788344 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.788346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.788966 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.788967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.789493 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:10.789494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.789802 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.789803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:10.789849 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.790278 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:10.790283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:10.792166 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
W20260925 19:10:10.792333 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:10.787795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50467 (local address 127.27.174.126:41403)
0925 19:10:10.787880 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:10.787882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:10.787945 (+ 63us) server_negotiation.cc:407] Connection header received
0925 19:10:10.788020 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:10.788020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:10.788032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:10.788072 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:10.788421 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.788897 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.789559 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:10.789731 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:10.789783 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:10.792255 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:50467: BlockingRecv error: recv got EOF from 127.27.174.67:50467 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:11.522449 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:11.517680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:48329)
0925 19:10:11.517868 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:11.517881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:11.517954 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:11.518119 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:11.518120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:11.518359 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:11.518422 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.518424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.519275 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.519276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.519816 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.519818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.520131 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.520132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.520169 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.520605 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:11.520609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:11.522392 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
W20260925 19:10:11.522548 5025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:11.517801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48329 (local address 127.27.174.126:41403)
0925 19:10:11.517949 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:11.517951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:11.517962 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:10:11.518013 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:11.518014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:11.518025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:11.518101 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:11.518551 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.519175 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.519887 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.520070 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.520122 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.522475 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:48329: BlockingRecv error: recv got EOF from 127.27.174.65:48329 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:11.653199 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:11.648530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:46907)
0925 19:10:11.648684 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:11.648696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:11.648782 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:11.648960 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:11.648961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:11.649132 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:11.649177 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.649179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.650020 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.650021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.650540 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.650542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.650894 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.650895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.650951 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.651396 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:11.651400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:11.653143 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
W20260925 19:10:11.653360 5025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:11.648608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:46907 (local address 127.27.174.126:41403)
0925 19:10:11.648674 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:11.648676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:11.648762 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:10:11.648860 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:11.648861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:11.648875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:11.648937 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:11.649252 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.649922 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.650616 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.650812 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.650893 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.653245 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:46907: BlockingRecv error: recv got EOF from 127.27.174.66:46907 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:11 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:11.796962 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:11.792785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:43575)
0925 19:10:11.792915 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:11.792925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:11.792989 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:11.793121 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:11.793121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:11.793263 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:11.793307 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.793308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.793878 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.793878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.794407 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:11.794409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.794712 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.794713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:11.794750 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.795144 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:11.795148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:11.796904 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
W20260925 19:10:11.797056 5025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:11.792838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:43575 (local address 127.27.174.126:41403)
0925 19:10:11.792891 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:11.792893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:11.792973 (+ 80us) server_negotiation.cc:407] Connection header received
0925 19:10:11.793035 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:11.793036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:11.793048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:11.793115 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:11.793357 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.793820 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.794479 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:11.794654 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:11.794703 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:11.796988 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:43575: BlockingRecv error: recv got EOF from 127.27.174.67:43575 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:12.527540 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:12.522892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:48805)
0925 19:10:12.523024 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:12.523035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:12.523097 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:12.523234 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:12.523235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:12.523421 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:12.523509 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.523512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.524212 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.524213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.524730 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.524731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.525095 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.525096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.525152 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.525623 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:12.525628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:12.527444 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
W20260925 19:10:12.527594 5029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:12.522957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48805 (local address 127.27.174.126:41403)
0925 19:10:12.523100 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:12.523103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:12.523112 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:10:12.523161 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:12.523162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:12.523174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:12.523231 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:12.523577 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.524124 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.524809 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.525013 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.525093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.527520 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:48805: BlockingRecv error: recv got EOF from 127.27.174.65:48805 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:12.658473 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:12.653731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:48241)
0925 19:10:12.653905 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:12.653919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:12.654013 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:12.654190 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:12.654191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:12.654375 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:12.654421 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.654423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.655093 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.655093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.655665 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.655666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.656012 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.656013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.656068 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.656564 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:12.656568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:12.658379 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
W20260925 19:10:12.658519 5029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:12.653822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48241 (local address 127.27.174.126:41403)
0925 19:10:12.653909 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:12.653912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:12.653992 (+ 80us) server_negotiation.cc:407] Connection header received
0925 19:10:12.654095 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:12.654096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:12.654108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:12.654185 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:12.654517 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.655015 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.655739 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.655933 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.656013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.658456 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:48241: BlockingRecv error: recv got EOF from 127.27.174.66:48241 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:12 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:12.801972 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:12.797476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:46455)
0925 19:10:12.797578 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:12.797590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:12.797679 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:12.797840 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:12.797840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:12.798059 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:12.798117 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.798119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.798755 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.798756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.799339 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:12.799341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.799669 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.799674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:12.799732 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.800180 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:12.800185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:12.801915 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
W20260925 19:10:12.802053 5029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:12.797503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:46455 (local address 127.27.174.126:41403)
0925 19:10:12.797565 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:12.797567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:12.797658 (+ 91us) server_negotiation.cc:407] Connection header received
0925 19:10:12.797746 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:12.797747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:12.797762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:12.797820 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:12.798183 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.798687 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.799409 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:12.799585 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:12.799654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:12.801992 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:46455: BlockingRecv error: recv got EOF from 127.27.174.67:46455 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:13.532821 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:13.528013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:45083)
0925 19:10:13.528154 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:13.528165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:13.528218 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:13.528358 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:13.528359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:13.528515 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:13.528562 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.528563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.529183 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.529184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.529696 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.529697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.530013 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.530014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.530052 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.530515 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:13.530521 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:13.532720 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260925 19:10:13.532874 5033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:13.528084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:45083 (local address 127.27.174.126:41403)
0925 19:10:13.528188 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:13.528189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:13.528202 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:10:13.528274 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:13.528275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:13.528284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:13.528343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:13.528621 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.529116 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.529764 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.529955 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.530006 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.532802 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:45083: BlockingRecv error: recv got EOF from 127.27.174.65:45083 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:13.664273 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:13.659013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:37813)
0925 19:10:13.659134 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:13.659150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:13.659210 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:13.659367 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:13.659367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:13.659538 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:13.659586 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.659588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.660338 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.660339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.661006 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.661010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.661660 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.661661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.661755 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.662202 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:13.662263 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:13.664195 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260925 19:10:13.664386 5033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:13.659142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37813 (local address 127.27.174.126:41403)
0925 19:10:13.659201 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:13.659202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:13.659217 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:10:13.659261 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:13.659262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:13.659281 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:13.659340 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:13.659727 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.660236 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.661206 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.661563 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.661660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.664293 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:37813: BlockingRecv error: recv got EOF from 127.27.174.66:37813 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:13 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:13.825820 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:13.802439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:49225)
0925 19:10:13.802558 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:13.802577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:13.802629 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:13.802740 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:13.802740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:13.802891 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:13.802938 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.802939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.803715 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.803716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.804291 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:13.804294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.804621 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.804622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:13.804662 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.824017 (+ 19355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:13.824022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:13.825755 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
W20260925 19:10:13.825910 5033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:13.802536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:49225 (local address 127.27.174.126:41403)
0925 19:10:13.802574 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:13.802576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:13.802615 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:10:13.802668 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:13.802669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:13.802679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:13.802726 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:13.803030 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.803557 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.804371 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:13.804552 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:13.804609 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:13.825846 (+ 21237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:49225: BlockingRecv error: recv got EOF from 127.27.174.67:49225 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:14.537676 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:14.533283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:57709)
0925 19:10:14.533401 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:14.533412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:14.533468 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:14.533619 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:14.533620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:14.533781 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:14.533827 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.533829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.534530 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.534530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.535040 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.535041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.535429 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.535430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.535469 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.535909 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:14.535914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:14.537608 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260925 19:10:14.537775 5037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:14.533339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57709 (local address 127.27.174.126:41403)
0925 19:10:14.533486 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:14.533488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:14.533498 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:14.533520 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:14.533521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:14.533531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:14.533607 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:14.533891 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.534456 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.535136 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.535358 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.535421 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.537703 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:57709: BlockingRecv error: recv got EOF from 127.27.174.65:57709 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:14.669425 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:14.664701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:40521)
0925 19:10:14.664886 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:14.664926 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:14.665023 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:14.665256 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:14.665257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:14.665434 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:14.665477 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.665479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.666131 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.666132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.666676 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.666678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.667049 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.667050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.667109 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.667543 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:14.667547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:14.669326 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
W20260925 19:10:14.669452 5037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:14.664778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:40521 (local address 127.27.174.126:41403)
0925 19:10:14.664861 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:14.664864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:14.664996 (+ 132us) server_negotiation.cc:407] Connection header received
0925 19:10:14.665174 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:14.665175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:14.665191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:14.665238 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:14.665549 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.666051 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.666766 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.666966 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.667048 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.669389 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:40521: BlockingRecv error: recv got EOF from 127.27.174.66:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:14 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:14.831156 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:14.826308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:50207)
0925 19:10:14.826512 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:14.826524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:14.826580 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:14.826729 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:14.826730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:14.826888 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:14.826935 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.826936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.827651 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.827653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.828222 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:14.828224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.828604 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.828605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:14.828655 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.829144 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:14.829148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:14.831071 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":126,"threads_started":1}
W20260925 19:10:14.831192 5037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:14.826423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50207 (local address 127.27.174.126:41403)
0925 19:10:14.826477 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:14.826478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:14.826566 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:10:14.826642 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:14.826643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:14.826652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:14.826709 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:14.826994 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.827481 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.828317 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:14.828512 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:14.828578 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:14.831142 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:50207: BlockingRecv error: recv got EOF from 127.27.174.67:50207 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:15.542795 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:15.538236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:47355)
0925 19:10:15.538385 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:15.538410 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:15.538483 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:15.538677 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:15.538678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:15.538884 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:15.538931 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.538933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.539567 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.539568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.540171 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.540174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.540499 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.540500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.540542 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.541051 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:15.541055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:15.542739 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
W20260925 19:10:15.542908 5041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:15.538310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47355 (local address 127.27.174.126:41403)
0925 19:10:15.538408 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:15.538411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:15.538480 (+ 69us) server_negotiation.cc:407] Connection header received
0925 19:10:15.538551 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:15.538552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:15.538568 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:15.538663 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:15.538999 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.539496 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.540246 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.540434 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.540487 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.542828 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:47355: BlockingRecv error: recv got EOF from 127.27.174.65:47355 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":45,"threads_started":1}
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:15.674598 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:15.669903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:45933)
0925 19:10:15.670026 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:15.670041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:15.670102 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:15.670271 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:15.670271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:15.670474 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:15.670521 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.670523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.671132 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.671133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.671787 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.671791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.672144 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.672146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.672193 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.672686 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:15.672691 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:15.674501 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
W20260925 19:10:15.674647 5041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:15.669999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45933 (local address 127.27.174.126:41403)
0925 19:10:15.670054 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:15.670056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:15.670083 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:10:15.670153 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:15.670154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:15.670196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:15.670252 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:15.670582 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.671062 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.671860 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.672046 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.672104 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.674582 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:45933: BlockingRecv error: recv got EOF from 127.27.174.66:45933 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:15 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:15.835888 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:15.831593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:44147)
0925 19:10:15.831737 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:15.831770 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:15.831834 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:15.831950 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:15.831950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:15.832112 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:15.832186 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.832188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.832792 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.832792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.833340 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:15.833341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.833655 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.833656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:15.833697 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.834110 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:15.834114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:15.835825 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260925 19:10:15.835958 5041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:15.831710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44147 (local address 127.27.174.126:41403)
0925 19:10:15.831776 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:15.831778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:15.831813 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:10:15.831877 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:15.831878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:15.831890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:15.831938 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:15.832237 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.832725 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.833413 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:15.833591 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:15.833644 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:15.835910 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:44147: BlockingRecv error: recv got EOF from 127.27.174.67:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:16.547976 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:16.543285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:50647)
0925 19:10:16.543420 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:16.543443 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:16.543503 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:16.543698 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:16.543699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:16.543862 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:16.543917 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.543919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.544610 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.544612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.545159 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.545162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.545524 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.545526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.545574 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.546035 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:16.546040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:16.547909 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260925 19:10:16.548086 5045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:16.543354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:50647 (local address 127.27.174.126:41403)
0925 19:10:16.543484 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:16.543486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:16.543502 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:10:16.543552 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:16.543552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:16.543562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:16.543638 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:16.543986 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.544521 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.545241 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.545434 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.545491 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.547992 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:50647: BlockingRecv error: recv got EOF from 127.27.174.65:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:16.679765 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:16.675111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:34299)
0925 19:10:16.675275 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:16.675285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:16.675336 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:16.675498 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:16.675499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:16.675741 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:16.675787 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.675789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.676418 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.676418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.676954 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.676956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.677308 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.677308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.677348 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.677783 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:16.677787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:16.679667 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260925 19:10:16.679915 5045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:16.675240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:34299 (local address 127.27.174.126:41403)
0925 19:10:16.675303 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:16.675305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:16.675320 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:10:16.675401 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:16.675402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:16.675412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:16.675457 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:16.675857 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.676349 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.677043 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.677241 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.677300 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.679774 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:34299: BlockingRecv error: recv got EOF from 127.27.174.66:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:16 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:16.841032 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:16.836363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:44585)
0925 19:10:16.836514 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:16.836527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:16.836596 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:16.836738 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:16.836738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:16.836921 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:16.836963 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.836965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.837696 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.837697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.838239 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:16.838241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.838598 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.838599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:16.838669 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.839166 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:16.839171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:16.840925 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
W20260925 19:10:16.841070 5045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:16.836479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44585 (local address 127.27.174.126:41403)
0925 19:10:16.836533 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:16.836535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:16.836576 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:10:16.836649 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:16.836650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:16.836664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:16.836729 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:16.837041 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.837627 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.838333 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:16.838520 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:16.838597 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:16.841003 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:44585: BlockingRecv error: recv got EOF from 127.27.174.67:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:17.553144 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:17.548468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:58339)
0925 19:10:17.548601 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:17.548612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:17.548679 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:17.548857 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:17.548858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:17.549029 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:17.549076 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.549078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.549783 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.549784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.550310 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.550312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.550714 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.550715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.550765 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.551283 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:17.551287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:17.553086 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260925 19:10:17.553339 5049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:17.548571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:58339 (local address 127.27.174.126:41403)
0925 19:10:17.548696 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:17.548698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:17.548715 (+ 17us) server_negotiation.cc:407] Connection header received
0925 19:10:17.548736 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:17.548737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:17.548746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:17.548807 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:17.549171 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.549677 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.550415 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.550609 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.550679 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.553248 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:58339: BlockingRecv error: recv got EOF from 127.27.174.65:58339 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:17.684793 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:17.680233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:47331)
0925 19:10:17.680388 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:17.680405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:17.680461 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:17.680597 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:17.680598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:17.680766 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:17.680812 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.680814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.681436 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.681437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.681974 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.681976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.682359 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.682360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.682400 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.682832 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:17.682836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:17.684725 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
W20260925 19:10:17.684906 5049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:17.680352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47331 (local address 127.27.174.126:41403)
0925 19:10:17.680413 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:17.680415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:17.680450 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:10:17.680524 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:17.680524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:17.680535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:17.680578 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:17.680875 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.681370 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.682066 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.682293 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.682352 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.684813 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:47331: BlockingRecv error: recv got EOF from 127.27.174.66:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:17 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:17.845708 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:17.841473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.67:44603)
0925 19:10:17.841605 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:17.841615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:17.841682 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:17.841840 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:17.841842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:17.841997 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:17.842044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.842046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.842658 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.842659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.843177 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:17.843178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.843476 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.843476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:17.843514 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.843966 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:17.843971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:17.845652 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260925 19:10:17.845870 5049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:17.841541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44603 (local address 127.27.174.126:41403)
0925 19:10:17.841598 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:17.841599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:17.841677 (+ 78us) server_negotiation.cc:407] Connection header received
0925 19:10:17.841723 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:17.841724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:17.841736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:17.841787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:17.842114 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.842592 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.843248 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:17.843418 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:17.843464 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:17.845773 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:44603: BlockingRecv error: recv got EOF from 127.27.174.67:44603 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 19:10:18 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:18 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:18.558166 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:18.553721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.65:60959)
0925 19:10:18.553838 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:18.553853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:18.553899 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:18.554034 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:18.554035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:18.554205 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:18.554253 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:18.554255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.554894 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.554895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:18.555432 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:18.555433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.555807 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.555808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:18.555852 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:18.556340 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:18.556344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:18.558100 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260925 19:10:18.558297 5053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:18.553776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:60959 (local address 127.27.174.126:41403)
0925 19:10:18.553898 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:18.553900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:18.553909 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:10:18.553945 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:18.553946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:18.553956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:18.554009 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:18.554317 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.554805 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.555504 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.555745 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.555798 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:18.558192 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:60959: BlockingRecv error: recv got EOF from 127.27.174.65:60959 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
Sep 25 19:10:18 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:10:18 dist-test-slave-fcdn krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:10:18.689666 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:18.685297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41403 (local address 127.27.174.66:51207)
0925 19:10:18.685463 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:18.685480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:18.685541 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:18.685668 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:18.685669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:18.685826 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:18.685873 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:18.685874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.686563 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.686564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:18.687101 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:18.687103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.687459 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.687460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:18.687498 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:18.687965 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:18.687969 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:18.689600 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41403: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260925 19:10:18.689765 5053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:18.685413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51207 (local address 127.27.174.126:41403)
0925 19:10:18.685463 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:18.685465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:18.685538 (+ 73us) server_negotiation.cc:407] Connection header received
0925 19:10:18.685588 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:18.685588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:18.685598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:18.685645 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:18.685982 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.686488 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.687199 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.687390 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.687454 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:18.689697 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:51207: BlockingRecv error: recv got EOF from 127.27.174.66:51207 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:10:18.744987 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4578
I20260925 19:10:18.749475 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4730
I20260925 19:10:18.753584 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4862
I20260925 19:10:18.757910 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 4508
2026-09-25T19:10:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17492 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T19:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:18Z Disabled control of system clock
I20260925 19:10:18.785461 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45975
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:10:18.865178 5058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:18.865383 5058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:18.865406 5058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:18.866571 5058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:18.866613 5058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:18.866626 5058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:18.866639 5058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:18.866688 5058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:18.868083 5058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:18.868327 5058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:18.868618 5058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:18.873489 5064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:18.873509 5066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:18.873509 5063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:18.873646 5058 server_base.cc:1034] running on GCE node
I20260925 19:10:18.874010 5058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:18.874359 5058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:18.875541 5058 hybrid_clock.cc:648] HybridClock initialized: now 1790363418875521 us; error 45 us; skew 500 ppm
I20260925 19:10:18.884155 5058 webserver.cc:460] Webserver started at http://127.27.174.126:33509/ using document root <none> and password file <none>
I20260925 19:10:18.884413 5058 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:18.884481 5058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:18.884649 5058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:18.885753 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "52e6edfc663142b3972abd93e8e99194"
format_stamp: "Formatted at 2026-09-25 19:10:18 on dist-test-slave-fcdn"
server_key: "864754b0dd92f797ae9cad3e1d0472a8"
server_key_iv: "a960be5fa1d1e567c011e3060589f352"
server_key_version: "encryptionkey@0"
I20260925 19:10:18.886116 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "52e6edfc663142b3972abd93e8e99194"
format_stamp: "Formatted at 2026-09-25 19:10:18 on dist-test-slave-fcdn"
server_key: "864754b0dd92f797ae9cad3e1d0472a8"
server_key_iv: "a960be5fa1d1e567c011e3060589f352"
server_key_version: "encryptionkey@0"
I20260925 19:10:18.887499 5058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:18.888326 5072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:18.888485 5058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:18.888584 5058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "52e6edfc663142b3972abd93e8e99194"
format_stamp: "Formatted at 2026-09-25 19:10:18 on dist-test-slave-fcdn"
server_key: "864754b0dd92f797ae9cad3e1d0472a8"
server_key_iv: "a960be5fa1d1e567c011e3060589f352"
server_key_version: "encryptionkey@0"
I20260925 19:10:18.888660 5058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:18.906029 5058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:18.906373 5058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:18.906526 5058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:18.910483 5058 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45975
I20260925 19:10:18.910517 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45975 every 8 connection(s)
I20260925 19:10:18.910861 5058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:10:18.911551 5125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:18.914209 5125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194: Bootstrap starting.
I20260925 19:10:18.914813 5125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:18.915102 5125 log.cc:824] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:18.915828 5125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194: No bootstrap required, opened a new log
I20260925 19:10:18.917367 5125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } }
I20260925 19:10:18.917507 5125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:18.917531 5125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e6edfc663142b3972abd93e8e99194, State: Initialized, Role: FOLLOWER
I20260925 19:10:18.917701 5125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } }
I20260925 19:10:18.917773 5125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:10:18.917822 5125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:10:18.917872 5125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:18.918411 5125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } }
I20260925 19:10:18.918517 5125 leader_election.cc:302] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e6edfc663142b3972abd93e8e99194; no voters:
I20260925 19:10:18.918680 5125 leader_election.cc:288] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:10:18.918745 5128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:10:18.918934 5128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [term 1 LEADER]: Becoming Leader. State: Replica: 52e6edfc663142b3972abd93e8e99194, State: Running, Role: LEADER
I20260925 19:10:18.918951 5125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:10:18.919071 5128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } }
I20260925 19:10:18.919425 5129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e6edfc663142b3972abd93e8e99194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } } }
I20260925 19:10:18.919580 5129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:18.919687 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5058
I20260925 19:10:18.919777 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:10:18.919956 28345 external_mini_cluster.cc:1428] Setting key ac6d7e9af7b8ddbd84b68714372e5882
I20260925 19:10:18.920113 5137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:10:18.920156 5130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e6edfc663142b3972abd93e8e99194. Latest consensus state: current_term: 1 leader_uuid: "52e6edfc663142b3972abd93e8e99194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e6edfc663142b3972abd93e8e99194" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45975 } } }
I20260925 19:10:18.920346 5130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:18.920707 5137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:10:18.922299 5137 catalog_manager.cc:1347] Generated new cluster ID: b2336af0edab4408aa76f6eccbf18f50
I20260925 19:10:18.922351 5137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:10:18.925544 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:18.920562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.27.174.126:45975)
0925 19:10:18.920986 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:18.920993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:18.921015 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:10:18.921275 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:18.921281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:18.921374 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:18.921510 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:18.921751 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.922862 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.923637 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:18.924049 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:18.924174 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:18.924868 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:18.924871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:18.924872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:18.925044 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:18.925075 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:18.925085 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:18.925170 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:18.925235 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:10:18.925394 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":92,"threads_started":1}
I20260925 19:10:18.944032 5137 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:10:18.944734 5137 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:10:18.953979 5137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e6edfc663142b3972abd93e8e99194: Generated new TSK 0
I20260925 19:10:18.954236 5137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:10:18.955907 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--builtin_ntp_servers=127.27.174.84:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:10:19.036993 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.037199 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.037218 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.038430 5149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.038470 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.038496 5149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:10:19.039880 5149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.040115 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.040412 5149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.046274 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.046238 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.046635 5149 server_base.cc:1034] running on GCE node
W20260925 19:10:19.046257 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.046900 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.047163 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.048318 5149 hybrid_clock.cc:648] HybridClock initialized: now 1790363419048306 us; error 27 us; skew 500 ppm
I20260925 19:10:19.049654 5149 webserver.cc:460] Webserver started at http://127.27.174.65:45069/ using document root <none> and password file <none>
I20260925 19:10:19.049898 5149 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.049952 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.050072 5149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.051429 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "84687f1f9b9446b38046272e4295e525"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "2dc5325c77550a24cad17fbc505585fb"
server_key_iv: "434f942d5f99fa2afc3879e055c45861"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.051844 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "84687f1f9b9446b38046272e4295e525"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "2dc5325c77550a24cad17fbc505585fb"
server_key_iv: "434f942d5f99fa2afc3879e055c45861"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.053375 5149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:19.054286 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.054476 5149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:19.054563 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "84687f1f9b9446b38046272e4295e525"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "2dc5325c77550a24cad17fbc505585fb"
server_key_iv: "434f942d5f99fa2afc3879e055c45861"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.054636 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.066222 5149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.066464 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.066560 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.066731 5149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.066982 5149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.067010 5149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.067031 5149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.067050 5149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.073215 5149 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40699
I20260925 19:10:19.073292 5276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40699 every 8 connection(s)
I20260925 19:10:19.073625 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:10:19.078802 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.074358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47487 (local address 127.27.174.126:45975)
0925 19:10:19.074399 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.074401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.074651 (+ 250us) server_negotiation.cc:407] Connection header received
0925 19:10:19.075145 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.075147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.075160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.075227 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.075986 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.076489 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.077472 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.077663 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.077713 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.078469 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.078470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.078471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.078475 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.078490 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.078494 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.078543 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.078710 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.078744 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:10:19.079002 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.074338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45975 (local address 127.27.174.65:47487)
0925 19:10:19.074518 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.074558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.075038 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.075326 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.075330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.075338 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.075907 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.075912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.076570 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.076571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.077396 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.077398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.077733 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.077733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.077841 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.078343 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.078343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.078370 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.078393 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.078589 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.078595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.078598 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.078679 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.078763 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260925 19:10:19.079515 5277 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45975
I20260925 19:10:19.079653 5277 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.079829 5277 heartbeater.cc:511] Master 127.27.174.126:45975 requested a full tablet report, sending...
I20260925 19:10:19.080307 5089 ts_manager.cc:194] Registered new tserver with Master: 84687f1f9b9446b38046272e4295e525 (127.27.174.65:40699)
I20260925 19:10:19.080825 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5149
I20260925 19:10:19.080916 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:10:19.081095 28345 external_mini_cluster.cc:1428] Setting key 07ef18765d7f200ee0fb55967a7fafd1
I20260925 19:10:19.081190 5089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:47487
I20260925 19:10:19.081988 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--builtin_ntp_servers=127.27.174.84:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:10:19.161410 5280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.161613 5280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.161633 5280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.162695 5280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.162727 5280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.162750 5280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:10:19.163990 5280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.164172 5280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.164402 5280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.169240 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.169289 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.169340 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.169377 5280 server_base.cc:1034] running on GCE node
I20260925 19:10:19.169687 5280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.169915 5280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.171068 5280 hybrid_clock.cc:648] HybridClock initialized: now 1790363419171058 us; error 31 us; skew 500 ppm
I20260925 19:10:19.172377 5280 webserver.cc:460] Webserver started at http://127.27.174.66:36499/ using document root <none> and password file <none>
I20260925 19:10:19.172616 5280 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.172698 5280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.172835 5280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.173857 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "03254345ccd4440d9d1c5285b8cc31de"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c992db463ad1e82e66c6f743dade1742"
server_key_iv: "89df4a96bce93c458b50b6489033dcdd"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.174225 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "03254345ccd4440d9d1c5285b8cc31de"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c992db463ad1e82e66c6f743dade1742"
server_key_iv: "89df4a96bce93c458b50b6489033dcdd"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.175511 5280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:19.176266 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.176431 5280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:10:19.176522 5280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "03254345ccd4440d9d1c5285b8cc31de"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c992db463ad1e82e66c6f743dade1742"
server_key_iv: "89df4a96bce93c458b50b6489033dcdd"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.176601 5280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.188884 5280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.189179 5280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.189321 5280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.189527 5280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.189846 5280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.189898 5280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.189939 5280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.189965 5280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.195739 5280 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:40261
I20260925 19:10:19.195835 5407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:40261 every 8 connection(s)
I20260925 19:10:19.196133 5280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:10:19.196446 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5280
I20260925 19:10:19.196543 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:10:19.196779 28345 external_mini_cluster.cc:1428] Setting key e3b8f16c10fbc2044cecdd69f0f43d68
I20260925 19:10:19.197790 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--builtin_ntp_servers=127.27.174.84:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 19:10:19.202176 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.196912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47131 (local address 127.27.174.126:45975)
0925 19:10:19.196964 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.196966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.197303 (+ 337us) server_negotiation.cc:407] Connection header received
0925 19:10:19.197934 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.197935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.197946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.198014 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.198736 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.199657 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.200594 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.200958 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.201041 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.201835 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.201836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.201837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.201842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.201862 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.201868 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.201944 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.201994 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.202028 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:10:19.202245 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.196921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45975 (local address 127.27.174.66:47131)
0925 19:10:19.197167 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.197215 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.197823 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.198108 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.198112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.198121 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.198641 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.198645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.199753 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.199754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.200512 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.200514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.201042 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.201043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.201148 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.201699 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.201699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.201728 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.201750 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.201941 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.201943 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.201946 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.201991 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.202065 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260925 19:10:19.202703 5408 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45975
I20260925 19:10:19.202814 5408 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.203049 5408 heartbeater.cc:511] Master 127.27.174.126:45975 requested a full tablet report, sending...
I20260925 19:10:19.203415 5089 ts_manager.cc:194] Registered new tserver with Master: 03254345ccd4440d9d1c5285b8cc31de (127.27.174.66:40261)
I20260925 19:10:19.203823 5089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:47131
W20260925 19:10:19.286239 5411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.286443 5411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.286463 5411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.287650 5411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.287696 5411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.287724 5411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:10:19.289048 5411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.289292 5411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.289579 5411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.294796 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.294878 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.294777 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.294929 5411 server_base.cc:1034] running on GCE node
I20260925 19:10:19.295389 5411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.295590 5411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.296777 5411 hybrid_clock.cc:648] HybridClock initialized: now 1790363419296763 us; error 30 us; skew 500 ppm
I20260925 19:10:19.298118 5411 webserver.cc:460] Webserver started at http://127.27.174.67:37785/ using document root <none> and password file <none>
I20260925 19:10:19.298350 5411 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.298398 5411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.298535 5411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.299588 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "25752d0ae1f341bcb42851403c06ab79"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "64dedefdca5f9067b835c94b1bbb3577"
server_key_iv: "61051b21e99cdd4054048cfd8d31797a"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.299994 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "25752d0ae1f341bcb42851403c06ab79"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "64dedefdca5f9067b835c94b1bbb3577"
server_key_iv: "61051b21e99cdd4054048cfd8d31797a"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.301220 5411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:19.301908 5425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.302143 5411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:19.302214 5411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "25752d0ae1f341bcb42851403c06ab79"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "64dedefdca5f9067b835c94b1bbb3577"
server_key_iv: "61051b21e99cdd4054048cfd8d31797a"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.302299 5411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.312796 5411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.313091 5411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.313226 5411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.313462 5411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.313786 5411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.313838 5411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.313879 5411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.313905 5411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.319516 5411 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:33675
I20260925 19:10:19.319571 5538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:33675 every 8 connection(s)
I20260925 19:10:19.319931 5411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:10:19.322968 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5411
I20260925 19:10:19.323048 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:10:19.323223 28345 external_mini_cluster.cc:1428] Setting key 4ef4f4d7e075ba4d921fe36131911f5d
I20260925 19:10:19.325174 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.320686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:60151 (local address 127.27.174.126:45975)
0925 19:10:19.320754 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.320756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.321000 (+ 244us) server_negotiation.cc:407] Connection header received
0925 19:10:19.321464 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.321465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.321475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.321540 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.322158 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.322634 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.323802 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.324084 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.324150 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.324956 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.324957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.324957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.324961 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.324975 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.324978 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.325021 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.325087 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.325117 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:10:19.325402 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.320695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45975 (local address 127.27.174.67:60151)
0925 19:10:19.320874 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.320915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.321386 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.321596 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.321599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.321607 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.322091 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.322095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.322705 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.322705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.323713 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:19.323716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:19.324218 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:19.324220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:19.324367 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:19.324832 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.324832 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.324859 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.324881 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.325031 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.325034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.325036 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.325087 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.325163 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 19:10:19.325840 5539 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45975
I20260925 19:10:19.325910 5539 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.326038 5539 heartbeater.cc:511] Master 127.27.174.126:45975 requested a full tablet report, sending...
I20260925 19:10:19.326336 5089 ts_manager.cc:194] Registered new tserver with Master: 25752d0ae1f341bcb42851403c06ab79 (127.27.174.67:33675)
I20260925 19:10:19.326696 5089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:60151
I20260925 19:10:19.335111 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:10:19.336910 5147 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:19.336482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.27.174.126:45975)
0925 19:10:19.336546 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.336548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.336648 (+ 100us) server_negotiation.cc:407] Connection header received
0925 19:10:19.336691 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.336692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.336702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.336754 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.336861 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59568: BlockingRecv error: recv got EOF from 127.0.0.1:59568 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:10:19.336975 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5149
I20260925 19:10:19.342859 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5280
I20260925 19:10:19.347110 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5411
I20260925 19:10:19.351146 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5058
2026-09-25T19:10:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (592 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T19:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:19Z Disabled control of system clock
I20260925 19:10:19.377298 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46835
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:10:19.456596 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.456807 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.456830 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.457918 5550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:19.457953 5550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.457966 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.457978 5550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:19.457990 5550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:19.459297 5550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.459708 5550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.459957 5550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.464689 5556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.464697 5558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.464920 5550 server_base.cc:1034] running on GCE node
W20260925 19:10:19.464708 5555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.465198 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.465498 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.466655 5550 hybrid_clock.cc:648] HybridClock initialized: now 1790363419466646 us; error 42 us; skew 500 ppm
I20260925 19:10:19.467916 5550 webserver.cc:460] Webserver started at http://127.27.174.126:46749/ using document root <none> and password file <none>
I20260925 19:10:19.468144 5550 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.468210 5550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.468361 5550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.469441 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "d44799a81e46411e95990978fc5a305e"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "477a5d99ed58961ca28a8355a68283c1"
server_key_iv: "b8ce981350861fb4c9426eb99b55a309"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.469816 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "d44799a81e46411e95990978fc5a305e"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "477a5d99ed58961ca28a8355a68283c1"
server_key_iv: "b8ce981350861fb4c9426eb99b55a309"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.471119 5550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:19.471886 5564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.472048 5550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:19.472141 5550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "d44799a81e46411e95990978fc5a305e"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "477a5d99ed58961ca28a8355a68283c1"
server_key_iv: "b8ce981350861fb4c9426eb99b55a309"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.472227 5550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.475456 5550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.475751 5550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.475896 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.479862 5550 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46835
I20260925 19:10:19.479903 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46835 every 8 connection(s)
I20260925 19:10:19.480233 5550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:10:19.480890 5617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:19.481191 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5550
I20260925 19:10:19.481294 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:10:19.481472 28345 external_mini_cluster.cc:1428] Setting key 6d5077b3c772bc3688a0a97f8ca8a9eb
I20260925 19:10:19.483965 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.481995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.27.174.126:46835)
0925 19:10:19.482215 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.482225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.482258 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:10:19.482701 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.482710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.482766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.482913 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.483527 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.483530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.483531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.483535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.483578 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.483592 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.483685 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.483738 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.483792 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260925 19:10:19.484941 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e: Bootstrap starting.
I20260925 19:10:19.485453 5617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:19.485682 5617 log.cc:824] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:19.486280 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e: No bootstrap required, opened a new log
I20260925 19:10:19.487540 5617 raft_consensus.cc:348] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } }
I20260925 19:10:19.487735 5617 raft_consensus.cc:374] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:19.487763 5617 raft_consensus.cc:729] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44799a81e46411e95990978fc5a305e, State: Initialized, Role: FOLLOWER
I20260925 19:10:19.487882 5617 consensus_queue.cc:260] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } }
I20260925 19:10:19.487955 5617 raft_consensus.cc:388] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:10:19.487977 5617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:10:19.488013 5617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:19.488476 5617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } }
I20260925 19:10:19.488565 5617 leader_election.cc:302] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d44799a81e46411e95990978fc5a305e; no voters:
I20260925 19:10:19.488683 5617 leader_election.cc:288] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:10:19.488955 5617 sys_catalog.cc:564] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:10:19.488976 5622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:10:19.489207 5622 raft_consensus.cc:686] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [term 1 LEADER]: Becoming Leader. State: Replica: d44799a81e46411e95990978fc5a305e, State: Running, Role: LEADER
I20260925 19:10:19.489296 5622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } }
I20260925 19:10:19.489656 5623 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d44799a81e46411e95990978fc5a305e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } } }
I20260925 19:10:19.489758 5623 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:19.489914 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d44799a81e46411e95990978fc5a305e. Latest consensus state: current_term: 1 leader_uuid: "d44799a81e46411e95990978fc5a305e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44799a81e46411e95990978fc5a305e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46835 } } }
I20260925 19:10:19.489964 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:19.489971 5631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:10:19.490685 5631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:10:19.492295 5631 catalog_manager.cc:1347] Generated new cluster ID: 755755a6f2c042fd944c5ed9f3981956
I20260925 19:10:19.492350 5631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:10:19.510891 5631 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:10:19.511075 5631 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:10:19.514681 5631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d44799a81e46411e95990978fc5a305e: Generated new TSK 0
I20260925 19:10:19.514856 5631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:10:19.518283 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--builtin_ntp_servers=127.27.174.84:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:10:19.600136 5641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.600376 5641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.600414 5641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.601594 5641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.601656 5641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.601699 5641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:10:19.603048 5641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.603308 5641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.603641 5641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.609354 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.609350 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.609351 5646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.609951 5641 server_base.cc:1034] running on GCE node
I20260925 19:10:19.610163 5641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.610409 5641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.611644 5641 hybrid_clock.cc:648] HybridClock initialized: now 1790363419611628 us; error 27 us; skew 500 ppm
I20260925 19:10:19.613137 5641 webserver.cc:460] Webserver started at http://127.27.174.65:36805/ using document root <none> and password file <none>
I20260925 19:10:19.613425 5641 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.613503 5641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.613685 5641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.615259 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "d5758094bc804a0ca58fcf6156609be6"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "35d21c784c17dcc9cd0f7420f35c46fa"
server_key_iv: "f810103b581d9cd6490970ab3dd7922f"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.615773 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "d5758094bc804a0ca58fcf6156609be6"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "35d21c784c17dcc9cd0f7420f35c46fa"
server_key_iv: "f810103b581d9cd6490970ab3dd7922f"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.617434 5641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:19.618251 5655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.618445 5641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:19.618553 5641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "d5758094bc804a0ca58fcf6156609be6"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "35d21c784c17dcc9cd0f7420f35c46fa"
server_key_iv: "f810103b581d9cd6490970ab3dd7922f"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.618626 5641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.621894 5641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.622165 5641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.622318 5641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.622519 5641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.622877 5641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.622941 5641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.622982 5641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.623008 5641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.628482 5641 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35783
I20260925 19:10:19.628561 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35783 every 8 connection(s)
I20260925 19:10:19.628906 5641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:10:19.631987 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.629618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33569 (local address 127.27.174.126:46835)
0925 19:10:19.629661 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.629663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.629894 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:10:19.630374 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.630375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.630381 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.630446 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.631778 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.631779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.631780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.631784 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.631799 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.631802 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.631854 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.631924 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.631935 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:10:19.632170 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.629608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46835 (local address 127.27.174.65:33569)
0925 19:10:19.629777 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.629808 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.630254 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.630500 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.630503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.630511 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.631587 (+ 1076us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.631587 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.631635 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.631667 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.631859 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.631862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.631864 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.631915 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.631960 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260925 19:10:19.632850 5769 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46835
I20260925 19:10:19.632956 5769 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.633116 5769 heartbeater.cc:511] Master 127.27.174.126:46835 requested a full tablet report, sending...
I20260925 19:10:19.633132 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5641
I20260925 19:10:19.633224 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:10:19.633402 28345 external_mini_cluster.cc:1428] Setting key 1ff83652663df6e3e7255e0ad9766cd0
I20260925 19:10:19.633703 5581 ts_manager.cc:194] Registered new tserver with Master: d5758094bc804a0ca58fcf6156609be6 (127.27.174.65:35783)
I20260925 19:10:19.634632 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--builtin_ntp_servers=127.27.174.84:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:10:19.716177 5772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.716398 5772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.716420 5772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.717648 5772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.717696 5772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.717722 5772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:10:19.719381 5772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.719761 5772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.720075 5772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.725613 5780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.725621 5778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.725795 5777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.725814 5772 server_base.cc:1034] running on GCE node
I20260925 19:10:19.726105 5772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.726308 5772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.727480 5772 hybrid_clock.cc:648] HybridClock initialized: now 1790363419727472 us; error 40 us; skew 500 ppm
I20260925 19:10:19.728996 5772 webserver.cc:460] Webserver started at http://127.27.174.66:45107/ using document root <none> and password file <none>
I20260925 19:10:19.729261 5772 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.729336 5772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.729491 5772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.730614 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "d800c9384a234ed5a891de5bcb5d7351"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "91af7cfc5b58eb6360f193344e102505"
server_key_iv: "c9eee19d7502263d228d0a4463e792f5"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.730990 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "d800c9384a234ed5a891de5bcb5d7351"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "91af7cfc5b58eb6360f193344e102505"
server_key_iv: "c9eee19d7502263d228d0a4463e792f5"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.732432 5772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:10:19.733331 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.733546 5772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:19.733657 5772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "d800c9384a234ed5a891de5bcb5d7351"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "91af7cfc5b58eb6360f193344e102505"
server_key_iv: "c9eee19d7502263d228d0a4463e792f5"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.733734 5772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.737260 5772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.737538 5772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.737679 5772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.737890 5772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.738191 5772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.738247 5772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.738287 5772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.738313 5772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.744774 5772 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:42627
I20260925 19:10:19.744866 5899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:42627 every 8 connection(s)
I20260925 19:10:19.745209 5772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:10:19.748459 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.745991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45449 (local address 127.27.174.126:46835)
0925 19:10:19.746074 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.746076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.746337 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:10:19.746883 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.746884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.746891 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.746944 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.748195 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.748197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.748197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.748203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.748220 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.748225 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.748273 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.748369 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.748384 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 19:10:19.748716 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.745973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46835 (local address 127.27.174.66:45449)
0925 19:10:19.746205 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.746241 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.746764 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.747038 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.747042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.747051 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.748036 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.748036 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.748068 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.748097 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.748282 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.748286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.748289 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.748380 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.748442 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":119,"threads_started":1}
I20260925 19:10:19.749226 5900 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46835
I20260925 19:10:19.749336 5900 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.749363 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5772
I20260925 19:10:19.749450 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:10:19.749490 5900 heartbeater.cc:511] Master 127.27.174.126:46835 requested a full tablet report, sending...
I20260925 19:10:19.749655 28345 external_mini_cluster.cc:1428] Setting key bb8556d67172c1494adbb91e643a0f2f
I20260925 19:10:19.749910 5580 ts_manager.cc:194] Registered new tserver with Master: d800c9384a234ed5a891de5bcb5d7351 (127.27.174.66:42627)
I20260925 19:10:19.750593 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--builtin_ntp_servers=127.27.174.84:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:10:19.837662 5903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:19.837932 5903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:19.837967 5903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:19.839530 5903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:19.839572 5903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:19.839648 5903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:10:19.840997 5903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:19.841192 5903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:19.841434 5903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:19.846366 5909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.846419 5911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:19.846390 5908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:19.846714 5903 server_base.cc:1034] running on GCE node
I20260925 19:10:19.846959 5903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:19.847256 5903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:19.848409 5903 hybrid_clock.cc:648] HybridClock initialized: now 1790363419848392 us; error 35 us; skew 500 ppm
I20260925 19:10:19.849712 5903 webserver.cc:460] Webserver started at http://127.27.174.67:40697/ using document root <none> and password file <none>
I20260925 19:10:19.849964 5903 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:19.850010 5903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:19.850162 5903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:19.851297 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "f86ad6fd804f4f36b836983651b50151"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c1fb3929751df945243b2fcf5286b6e4"
server_key_iv: "a5a1fb29fd9bc73dd029409aeabb3117"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.851706 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "f86ad6fd804f4f36b836983651b50151"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c1fb3929751df945243b2fcf5286b6e4"
server_key_iv: "a5a1fb29fd9bc73dd029409aeabb3117"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.852998 5903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:19.853711 5917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.853857 5903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:19.853940 5903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "f86ad6fd804f4f36b836983651b50151"
format_stamp: "Formatted at 2026-09-25 19:10:19 on dist-test-slave-fcdn"
server_key: "c1fb3929751df945243b2fcf5286b6e4"
server_key_iv: "a5a1fb29fd9bc73dd029409aeabb3117"
server_key_version: "encryptionkey@0"
I20260925 19:10:19.854025 5903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:19.857434 5903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:19.857703 5903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:19.857841 5903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:19.858047 5903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:19.858358 5903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:19.858390 5903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.858412 5903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:19.858457 5903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:19.864033 5903 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:38349
I20260925 19:10:19.864120 6030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:38349 every 8 connection(s)
I20260925 19:10:19.864444 5903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:10:19.867368 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.865188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35589 (local address 127.27.174.126:46835)
0925 19:10:19.865258 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.865260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.865487 (+ 227us) server_negotiation.cc:407] Connection header received
0925 19:10:19.865934 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.865935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.865940 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.866005 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.867152 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:19.867153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:19.867154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:10:19.867157 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:19.867170 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:10:19.867174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:19.867224 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:19.867304 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 19:10:19.867317 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:10:19.867574 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:19.865160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46835 (local address 127.27.174.67:35589)
0925 19:10:19.865352 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:19.865393 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:19.865849 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:19.866078 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:19.866081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:19.866089 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:19.867001 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:10:19.867001 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:19.867030 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:10:19.867059 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:19.867234 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:19.867237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:19.867239 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:10:19.867295 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:10:19.867345 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260925 19:10:19.868149 6031 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46835
I20260925 19:10:19.868232 6031 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:19.868342 6031 heartbeater.cc:511] Master 127.27.174.126:46835 requested a full tablet report, sending...
I20260925 19:10:19.868708 5581 ts_manager.cc:194] Registered new tserver with Master: f86ad6fd804f4f36b836983651b50151 (127.27.174.67:38349)
I20260925 19:10:19.870779 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 5903
I20260925 19:10:19.870851 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:10:19.871048 28345 external_mini_cluster.cc:1428] Setting key ebd113035f37d36f0e1105e578ac9cce
I20260925 19:10:19.872053 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:10:19.874264 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5641
W20260925 19:10:19.874322 5620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:10:19.873516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.27.174.126:46835)
0925 19:10:19.873590 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:19.873592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:19.873726 (+ 134us) server_negotiation.cc:407] Connection header received
0925 19:10:19.873819 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:19.873820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:19.873827 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:19.873922 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:19.874183 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33122: BlockingRecv error: recv got EOF from 127.0.0.1:33122 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:10:19.879568 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5772
I20260925 19:10:19.884110 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5903
I20260925 19:10:19.888221 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 5550
2026-09-25T19:10:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (537 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:10:19 dist-test-slave-fcdn krb5kdc[6041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:10:19 dist-test-slave-fcdn krb5kdc[6041](info): set up 2 sockets
Sep 25 19:10:19 dist-test-slave-fcdn krb5kdc[6041](info): commencing operation
krb5kdc: starting...
W20260925 19:10:20.920513 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:10:20 dist-test-slave-fcdn krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:10:21.039376 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44043
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:21.118196 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:21.118395 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:21.118418 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:21.119485 6054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:10:21.119519 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:21.119534 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:21.119545 6054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:10:21.119558 6054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:10:21.121021 6054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:21.121212 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:21.121373 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:21.126185 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.126200 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.126206 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:21.126525 6054 server_base.cc:1034] running on GCE node
I20260925 19:10:21.126791 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:21.127081 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:21.128283 6054 hybrid_clock.cc:648] HybridClock initialized: now 1790363421128269 us; error 32 us; skew 500 ppm
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:21.130733 6054 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:10:21.131224 6054 webserver.cc:460] Webserver started at http://127.27.174.126:35799/ using document root <none> and password file <none>
I20260925 19:10:21.131438 6054 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:21.131498 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:21.131685 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:21.132740 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "3baf1458320f4c47876b21e9cc521a48"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "7f27505380ee80d772bb43ffa0528181"
server_key_iv: "85155f2c1302ea53496cc6094cfeb7c7"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.133100 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "3baf1458320f4c47876b21e9cc521a48"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "7f27505380ee80d772bb43ffa0528181"
server_key_iv: "85155f2c1302ea53496cc6094cfeb7c7"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.134465 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:10:21.135320 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.135542 6054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:21.135680 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "3baf1458320f4c47876b21e9cc521a48"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "7f27505380ee80d772bb43ffa0528181"
server_key_iv: "85155f2c1302ea53496cc6094cfeb7c7"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.135740 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:21.143133 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:21.145031 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:21.145153 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:21.148842 6054 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44043
I20260925 19:10:21.148885 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44043 every 8 connection(s)
I20260925 19:10:21.149217 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:10:21.150177 6122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:21.152518 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: Bootstrap starting.
I20260925 19:10:21.153090 6122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:21.153366 6122 log.cc:824] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:21.153947 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 6054
I20260925 19:10:21.154079 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:10:21.154151 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: No bootstrap required, opened a new log
I20260925 19:10:21.154256 28345 external_mini_cluster.cc:1428] Setting key 550d7a79aac4aafd589169d58a78abab
I20260925 19:10:21.156026 6122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } }
I20260925 19:10:21.156185 6122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:21.156229 6122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baf1458320f4c47876b21e9cc521a48, State: Initialized, Role: FOLLOWER
I20260925 19:10:21.156349 6122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } }
I20260925 19:10:21.156440 6122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:10:21.156484 6122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:10:21.156554 6122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:21.157424 6122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } }
I20260925 19:10:21.157541 6122 leader_election.cc:302] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3baf1458320f4c47876b21e9cc521a48; no voters:
I20260925 19:10:21.157729 6122 leader_election.cc:288] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:10:21.157749 6127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:10:21.157861 6127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [term 1 LEADER]: Becoming Leader. State: Replica: 3baf1458320f4c47876b21e9cc521a48, State: Running, Role: LEADER
I20260925 19:10:21.157941 6127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } }
I20260925 19:10:21.157987 6122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:10:21.158267 6128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3baf1458320f4c47876b21e9cc521a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } } }
I20260925 19:10:21.158385 6128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:21.158263 6129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3baf1458320f4c47876b21e9cc521a48. Latest consensus state: current_term: 1 leader_uuid: "3baf1458320f4c47876b21e9cc521a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baf1458320f4c47876b21e9cc521a48" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44043 } } }
I20260925 19:10:21.158631 6129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48 [sys.catalog]: This master's current role is: LEADER
I20260925 19:10:21.159198 6133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:10:21.159889 6133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:10:21.161273 6133 catalog_manager.cc:1347] Generated new cluster ID: b273755c1c674aa7ba49a5d9d521f889
I20260925 19:10:21.161322 6133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:21.166360 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.154782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.27.174.126:44043)
0925 19:10:21.154988 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.154998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.155031 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:10:21.156006 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.156013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.156060 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.156209 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:21.156942 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.158018 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.159087 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.159785 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.159894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.162678 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:21.162682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:21.162682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:21.162695 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:21.164712 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.165100 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.165103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.165104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.165229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.165434 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.165435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.165435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.165735 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:21.165889 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.165989 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.166158 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260925 19:10:21.172695 6133 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:10:21.173184 6133 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:10:21.178130 6133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: Generated new TSK 0
I20260925 19:10:21.178300 6133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:10:21.206293 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:21.288352 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:21.288555 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:21.288579 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:21.289734 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:21.289779 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:21.289805 6148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:10:21.291332 6148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:21.291515 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:21.291792 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:21.296698 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.296747 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.296747 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:21.296898 6148 server_base.cc:1034] running on GCE node
I20260925 19:10:21.297125 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:21.297360 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:21.298519 6148 hybrid_clock.cc:648] HybridClock initialized: now 1790363421298509 us; error 30 us; skew 500 ppm
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:21.301156 6148 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:10:21.301607 6148 webserver.cc:460] Webserver started at http://127.27.174.65:42985/ using document root <none> and password file <none>
I20260925 19:10:21.301816 6148 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:21.301859 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:21.301949 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:21.302940 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "fb9f30e8a07e4828b41bee172bcbda36"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "ed44cb456c0d61268692e321cb2a160f"
server_key_iv: "0890b42bafd7b32313eed46c031ab5ed"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.303236 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "fb9f30e8a07e4828b41bee172bcbda36"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "ed44cb456c0d61268692e321cb2a160f"
server_key_iv: "0890b42bafd7b32313eed46c031ab5ed"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.304562 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:10:21.305290 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.305495 6148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:10:21.305588 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "fb9f30e8a07e4828b41bee172bcbda36"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "ed44cb456c0d61268692e321cb2a160f"
server_key_iv: "0890b42bafd7b32313eed46c031ab5ed"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.305647 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:21.322296 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:21.324472 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:21.324589 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:21.324787 6148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:21.325114 6148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:21.325150 6148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.325172 6148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:21.325214 6148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.331137 6148 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37883
I20260925 19:10:21.331210 6276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37883 every 8 connection(s)
I20260925 19:10:21.331545 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:21.340718 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.332337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:59249 (local address 127.27.174.126:44043)
0925 19:10:21.332411 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.332413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.332631 (+ 218us) server_negotiation.cc:407] Connection header received
0925 19:10:21.333134 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.333135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.333145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.333208 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:21.334110 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.334656 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.335663 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.335881 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.335938 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.338002 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:21.338003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:21.338003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:21.338012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:21.339516 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.339935 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.339936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.339936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.339988 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.340204 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.340205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.340205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.340310 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:21.340365 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.340584 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.340651 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:10:21.340859 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.332316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44043 (local address 127.27.174.65:59249)
0925 19:10:21.332501 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.332542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.333012 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.333260 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.333263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.333818 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:21.334018 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.334022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.334734 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.334735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.335534 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.335536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.335954 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.335954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.336070 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.336588 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:21.336596 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:21.337918 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:21.339657 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.339663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.339664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.339872 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.340047 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.340047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.340047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.340146 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.340366 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:21.340369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:21.340474 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.340552 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.340672 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260925 19:10:21.341238 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 6148
I20260925 19:10:21.341370 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:10:21.341349 6277 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44043
I20260925 19:10:21.341476 6277 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:21.341629 28345 external_mini_cluster.cc:1428] Setting key c76ee16f46274b0cacb8c90be1003c25
I20260925 19:10:21.341653 6277 heartbeater.cc:511] Master 127.27.174.126:44043 requested a full tablet report, sending...
I20260925 19:10:21.342157 6086 ts_manager.cc:194] Registered new tserver with Master: fb9f30e8a07e4828b41bee172bcbda36 (127.27.174.65:37883)
I20260925 19:10:21.342738 6086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:59249
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:10:21.366703 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:21.446139 6282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:21.446354 6282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:21.446391 6282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:21.447538 6282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:21.447602 6282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:21.447678 6282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:10:21.449260 6282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:21.449524 6282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:21.449756 6282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:21.454766 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.454735 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.454756 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:21.454905 6282 server_base.cc:1034] running on GCE node
I20260925 19:10:21.455138 6282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:21.455379 6282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:21.456538 6282 hybrid_clock.cc:648] HybridClock initialized: now 1790363421456513 us; error 45 us; skew 500 ppm
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:21.459115 6282 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:10:21.459559 6282 webserver.cc:460] Webserver started at http://127.27.174.66:36701/ using document root <none> and password file <none>
I20260925 19:10:21.459832 6282 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:21.459899 6282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:21.460035 6282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:21.461035 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "124efd19acb54d72b9a0da3d731af4bb"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "d779e20d83f0945e0a9edb38f586a987"
server_key_iv: "cbfb5c0bfd48d37c9e21a5626217f913"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.461388 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "124efd19acb54d72b9a0da3d731af4bb"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "d779e20d83f0945e0a9edb38f586a987"
server_key_iv: "cbfb5c0bfd48d37c9e21a5626217f913"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.462654 6282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:10:21.463366 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.463562 6282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:10:21.463680 6282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "124efd19acb54d72b9a0da3d731af4bb"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "d779e20d83f0945e0a9edb38f586a987"
server_key_iv: "cbfb5c0bfd48d37c9e21a5626217f913"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.463763 6282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:21.471349 6282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:21.473322 6282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:21.473477 6282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:21.473690 6282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:21.474000 6282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:21.474052 6282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.474098 6282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:21.474123 6282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.479769 6282 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:40757
I20260925 19:10:21.479822 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:40757 every 8 connection(s)
I20260925 19:10:21.480158 6282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:10:21.481108 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 6282
I20260925 19:10:21.481220 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:10:21.481415 28345 external_mini_cluster.cc:1428] Setting key fd53c827a9dabe7420b4f112dfac83ad
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20260925 19:10:21.490202 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.480871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:43825 (local address 127.27.174.126:44043)
0925 19:10:21.480930 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.480933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.481248 (+ 315us) server_negotiation.cc:407] Connection header received
0925 19:10:21.481891 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.481892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.481900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.481940 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:21.483218 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.483966 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.484934 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.485136 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.485223 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.487393 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:21.487398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:21.487399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:21.487405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:21.488789 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.489203 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.489204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.489205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.489264 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.489529 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.489531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.489531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.489684 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:21.489768 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.490059 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.490123 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:10:21.490351 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.480858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44043 (local address 127.27.174.66:43825)
0925 19:10:21.481083 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.481129 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.481795 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.482018 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.482022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.482841 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:21.483121 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.483127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.484051 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.484051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.484826 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.484828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.485229 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.485230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.485370 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.485947 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:21.485955 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:21.487286 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:21.488884 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.488890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.488892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.489113 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.489329 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.489330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.489330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.489440 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.489781 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:21.489787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:21.489902 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.490033 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.490153 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260925 19:10:21.490854 6411 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44043
I20260925 19:10:21.490968 6411 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:21.491137 6411 heartbeater.cc:511] Master 127.27.174.126:44043 requested a full tablet report, sending...
I20260925 19:10:21.491539 6086 ts_manager.cc:194] Registered new tserver with Master: 124efd19acb54d72b9a0da3d731af4bb (127.27.174.66:40757)
I20260925 19:10:21.491928 6086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:43825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:10:21.506598 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:10:21.584931 6416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:10:21.585156 6416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:10:21.585188 6416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:10:21.586277 6416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:10:21.586344 6416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:10:21.586387 6416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:10:21.587809 6416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:10:21.588063 6416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:10:21.588281 6416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:10:21.593448 6424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:21.593487 6416 server_base.cc:1034] running on GCE node
W20260925 19:10:21.593441 6422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:10:21.593441 6421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:10:21.593860 6416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:10:21.594059 6416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:10:21.595206 6416 hybrid_clock.cc:648] HybridClock initialized: now 1790363421595190 us; error 40 us; skew 500 ppm
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:10:21.597681 6416 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:10:21.598124 6416 webserver.cc:460] Webserver started at http://127.27.174.67:34807/ using document root <none> and password file <none>
I20260925 19:10:21.598361 6416 fs_manager.cc:359] Metadata directory not provided
I20260925 19:10:21.598405 6416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:10:21.598562 6416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:10:21.599558 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "24351300d33f4d22828222aa99632e2f"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "c44641855acf98ea0dde30762cf44abb"
server_key_iv: "77e4068e9a44af88b228e8af406ae017"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.599951 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "24351300d33f4d22828222aa99632e2f"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "c44641855acf98ea0dde30762cf44abb"
server_key_iv: "77e4068e9a44af88b228e8af406ae017"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.601214 6416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:10:21.601917 6431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.602105 6416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:10:21.602208 6416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "24351300d33f4d22828222aa99632e2f"
format_stamp: "Formatted at 2026-09-25 19:10:21 on dist-test-slave-fcdn"
server_key: "c44641855acf98ea0dde30762cf44abb"
server_key_iv: "77e4068e9a44af88b228e8af406ae017"
server_key_version: "encryptionkey@0"
I20260925 19:10:21.602279 6416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:10:21.616765 6416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:10:21.618525 6416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:10:21.618666 6416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:10:21.618886 6416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:10:21.619184 6416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:10:21.619216 6416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.619238 6416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:10:21.619282 6416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:10:21.624982 6416 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:35617
I20260925 19:10:21.625031 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:35617 every 8 connection(s)
I20260925 19:10:21.625319 6416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:10:21.631258 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 6416
I20260925 19:10:21.631348 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:10:21.631525 28345 external_mini_cluster.cc:1428] Setting key ee6c6baf70e5b2c027f41a5c06de6091
I20260925 19:10:21.634595 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.626012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:36105 (local address 127.27.174.126:44043)
0925 19:10:21.626069 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.626071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.626188 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:10:21.626802 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.626803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.626816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.626926 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:21.628104 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.628595 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.629574 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.629766 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.629837 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.632046 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:21.632047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:21.632048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:21.632055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:21.633526 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.633873 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.633874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.633874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.633924 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.634129 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.634129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.634130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.634230 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:21.634279 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.634475 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.634535 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:10:21.634778 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.625922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44043 (local address 127.27.174.67:36105)
0925 19:10:21.626075 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.626116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.626678 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.627014 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.627018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.627821 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:10:21.628000 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.628005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.628683 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.628684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.629469 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.629471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.629853 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.629854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.629989 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.630414 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:10:21.630422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:10:21.631957 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:10:21.633598 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.633603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.633605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.633809 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.633988 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:10:21.633989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:10:21.633989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:10:21.634076 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:10:21.634280 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:10:21.634284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:10:21.634384 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.634461 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.634569 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260925 19:10:21.635248 6545 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44043
I20260925 19:10:21.635340 6545 heartbeater.cc:464] Registering TS with master...
I20260925 19:10:21.635528 6545 heartbeater.cc:511] Master 127.27.174.126:44043 requested a full tablet report, sending...
I20260925 19:10:21.635891 6086 ts_manager.cc:194] Registered new tserver with Master: 24351300d33f4d22828222aa99632e2f (127.27.174.67:35617)
I20260925 19:10:21.636327 6086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:36105
I20260925 19:10:21.643195 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:10:21.650341 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.644532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.27.174.126:44043)
0925 19:10:21.644605 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.644607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.644712 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:10:21.644784 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.644785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.644795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.644861 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:10:21.645403 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.645859 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.646590 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.646775 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.646868 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.648043 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:10:21.648045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:10:21.648045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:10:21.648053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:10:21.649250 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.649575 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.649577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.649577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.649631 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:10:21.649822 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:10:21.649823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:10:21.649823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:10:21.649960 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:10:21.650059 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.650175 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.650246 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:10:21.651923 6086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:10:21.653403 6086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:10:21.659549 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:40757 (local address 127.0.0.1:58128)
0925 19:10:21.655486 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.655522 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.655590 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.656178 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.656179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.656181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:10:21.656234 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.656236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657138 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.657139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.658363 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.658365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.659278 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.659349 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.659362 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.659426 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.659483 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260925 19:10:21.659705 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.27.174.66:40757)
0925 19:10:21.656010 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.656013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.656023 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:21.656058 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.656059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.656074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.656308 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:10:21.656378 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657034 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.658438 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.659212 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659276 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.659520 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.659547 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.659587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260925 19:10:21.659997 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35617 (local address 127.0.0.1:46052)
0925 19:10:21.655815 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.655824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.655870 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.656079 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.656082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.656084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:10:21.656146 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.656148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657438 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.657439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.658494 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.658496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.659267 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.659349 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.659362 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.659432 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.659930 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":450,"thread_start_us":57,"threads_started":1}
I20260925 19:10:21.660117 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.27.174.67:35617)
0925 19:10:21.655772 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.655776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.655867 (+ 91us) server_negotiation.cc:407] Connection header received
0925 19:10:21.655956 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.655958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.655974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.656058 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:10:21.656233 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657339 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.658575 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.659084 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659169 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.659224 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.659473 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.659509 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260925 19:10:21.660266 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37883 (local address 127.0.0.1:56640)
0925 19:10:21.655681 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.655692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.655745 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.656387 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.656388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.656390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:10:21.656425 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.656426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657723 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.657724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.658996 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.658999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.660002 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.660003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.660080 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.660085 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.660141 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.660176 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260925 19:10:21.660380 6345 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 (DEFAULT_TABLE table=test-table [id=eca405be1cf54027b413a2b8b1f26217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:10:21.660512 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.655653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.27.174.65:37883)
0925 19:10:21.656069 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.656184 (+ 115us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.656194 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:21.656243 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.656245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.656267 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.656499 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:10:21.656567 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.657597 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659092 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.659833 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.659919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.659970 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.660359 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.660407 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":75,"threads_started":1}
I20260925 19:10:21.660727 6345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1bfd0e6ad34954b78a6d2575489e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:21.661125 6211 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 (DEFAULT_TABLE table=test-table [id=eca405be1cf54027b413a2b8b1f26217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:10:21.661434 6211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1bfd0e6ad34954b78a6d2575489e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:21.664073 6559 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Bootstrap starting.
I20260925 19:10:21.664208 6479 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 (DEFAULT_TABLE table=test-table [id=eca405be1cf54027b413a2b8b1f26217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:10:21.664507 6479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1bfd0e6ad34954b78a6d2575489e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:10:21.664701 6560 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: Bootstrap starting.
I20260925 19:10:21.664749 6559 tablet_bootstrap.cc:654] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:21.665212 6560 tablet_bootstrap.cc:654] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:21.665323 6559 log.cc:824] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:21.665649 6560 log.cc:824] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:21.666106 6559 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: No bootstrap required, opened a new log
I20260925 19:10:21.666169 6559 ts_tablet_manager.cc:1397] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:10:21.666368 6560 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: No bootstrap required, opened a new log
I20260925 19:10:21.666424 6560 ts_tablet_manager.cc:1397] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:10:21.667703 6563 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Bootstrap starting.
I20260925 19:10:21.668301 6563 tablet_bootstrap.cc:654] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Neither blocks nor log segments found. Creating new log.
I20260925 19:10:21.668172 6560 raft_consensus.cc:348] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.668349 6560 raft_consensus.cc:374] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:21.668385 6560 raft_consensus.cc:729] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f30e8a07e4828b41bee172bcbda36, State: Initialized, Role: FOLLOWER
I20260925 19:10:21.668339 6559 raft_consensus.cc:348] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.668519 6560 consensus_queue.cc:260] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.668582 6559 raft_consensus.cc:374] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:21.668630 6559 raft_consensus.cc:729] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124efd19acb54d72b9a0da3d731af4bb, State: Initialized, Role: FOLLOWER
I20260925 19:10:21.668771 6559 consensus_queue.cc:260] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.668826 6560 ts_tablet_manager.cc:1428] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:10:21.668830 6277 heartbeater.cc:503] Master 127.27.174.126:44043 was elected leader, sending a full tablet report...
I20260925 19:10:21.668895 6563 log.cc:824] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:10:21.669040 6411 heartbeater.cc:503] Master 127.27.174.126:44043 was elected leader, sending a full tablet report...
I20260925 19:10:21.669464 6559 ts_tablet_manager.cc:1428] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:10:21.669572 6563 tablet_bootstrap.cc:492] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: No bootstrap required, opened a new log
I20260925 19:10:21.669649 6563 ts_tablet_manager.cc:1397] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:10:21.671262 6563 raft_consensus.cc:348] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.671394 6563 raft_consensus.cc:374] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:10:21.671437 6563 raft_consensus.cc:729] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24351300d33f4d22828222aa99632e2f, State: Initialized, Role: FOLLOWER
I20260925 19:10:21.671542 6563 consensus_queue.cc:260] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.671806 6563 ts_tablet_manager.cc:1428] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:10:21.671861 6545 heartbeater.cc:503] Master 127.27.174.126:44043 was elected leader, sending a full tablet report...
W20260925 19:10:21.730721 6412 tablet.cc:2369] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:10:21.756055 6564 raft_consensus.cc:482] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:10:21.756224 6564 raft_consensus.cc:504] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.756625 6564 leader_election.cc:288] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124efd19acb54d72b9a0da3d731af4bb (127.27.174.66:40757), 24351300d33f4d22828222aa99632e2f (127.27.174.67:35617)
I20260925 19:10:21.759889 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.756983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49813 (local address 127.27.174.66:40757)
0925 19:10:21.757043 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.757044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.757234 (+ 190us) server_negotiation.cc:407] Connection header received
0925 19:10:21.757274 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.757275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.757288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.757337 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:10:21.757486 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.758010 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759132 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.759513 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759580 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.759588 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.759789 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.759829 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:10:21.759897 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.756838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:40757 (local address 127.27.174.65:49813)
0925 19:10:21.757158 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.757168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.757222 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.757354 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.757355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.757356 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:10:21.757424 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.757425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.758111 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.758111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.759038 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.759040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.759590 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.759685 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.759690 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.759772 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.759809 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":157,"thread_start_us":117,"threads_started":1}
I20260925 19:10:21.760062 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.756924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:53285 (local address 127.27.174.67:35617)
0925 19:10:21.756979 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.756981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.756991 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:10:21.757038 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.757039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.757050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.757125 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:10:21.757344 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.757950 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759140 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.759648 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.759750 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.759948 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.759996 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:10:21.760079 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.756838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35617 (local address 127.27.174.65:53285)
0925 19:10:21.756883 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.756895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:10:21.756940 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:10:21.757130 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:10:21.757132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:10:21.757134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:10:21.757226 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.757228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.758025 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.758026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.759035 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:10:21.759037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.759766 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.759767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:10:21.759855 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.759863 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:10:21.759981 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 19:10:21.760017 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":2}
I20260925 19:10:21.760313 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1bfd0e6ad34954b78a6d2575489e73" candidate_uuid: "fb9f30e8a07e4828b41bee172bcbda36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124efd19acb54d72b9a0da3d731af4bb" is_pre_election: true
I20260925 19:10:21.760381 6499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1bfd0e6ad34954b78a6d2575489e73" candidate_uuid: "fb9f30e8a07e4828b41bee172bcbda36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24351300d33f4d22828222aa99632e2f" is_pre_election: true
I20260925 19:10:21.760522 6365 raft_consensus.cc:2454] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9f30e8a07e4828b41bee172bcbda36 in term 0.
I20260925 19:10:21.760526 6499 raft_consensus.cc:2454] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9f30e8a07e4828b41bee172bcbda36 in term 0.
I20260925 19:10:21.760753 6165 leader_election.cc:302] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124efd19acb54d72b9a0da3d731af4bb, fb9f30e8a07e4828b41bee172bcbda36; no voters:
I20260925 19:10:21.760944 6564 raft_consensus.cc:2781] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:10:21.761022 6564 raft_consensus.cc:482] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:10:21.761057 6564 raft_consensus.cc:3037] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:21.761777 6564 raft_consensus.cc:504] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.761929 6564 leader_election.cc:288] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [CANDIDATE]: Term 1 election: Requested vote from peers 124efd19acb54d72b9a0da3d731af4bb (127.27.174.66:40757), 24351300d33f4d22828222aa99632e2f (127.27.174.67:35617)
I20260925 19:10:21.762076 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1bfd0e6ad34954b78a6d2575489e73" candidate_uuid: "fb9f30e8a07e4828b41bee172bcbda36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124efd19acb54d72b9a0da3d731af4bb"
I20260925 19:10:21.762178 6365 raft_consensus.cc:3037] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:21.762938 6365 raft_consensus.cc:2454] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9f30e8a07e4828b41bee172bcbda36 in term 1.
I20260925 19:10:21.763067 6499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1bfd0e6ad34954b78a6d2575489e73" candidate_uuid: "fb9f30e8a07e4828b41bee172bcbda36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24351300d33f4d22828222aa99632e2f"
I20260925 19:10:21.763132 6165 leader_election.cc:302] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124efd19acb54d72b9a0da3d731af4bb, fb9f30e8a07e4828b41bee172bcbda36; no voters:
I20260925 19:10:21.763166 6499 raft_consensus.cc:3037] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:10:21.763207 6564 raft_consensus.cc:2781] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:10:21.763347 6564 raft_consensus.cc:686] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [term 1 LEADER]: Becoming Leader. State: Replica: fb9f30e8a07e4828b41bee172bcbda36, State: Running, Role: LEADER
I20260925 19:10:21.763454 6564 consensus_queue.cc:237] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } }
I20260925 19:10:21.764112 6499 raft_consensus.cc:2454] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9f30e8a07e4828b41bee172bcbda36 in term 1.
I20260925 19:10:21.764325 6085 catalog_manager.cc:5495] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb9f30e8a07e4828b41bee172bcbda36 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "fb9f30e8a07e4828b41bee172bcbda36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f30e8a07e4828b41bee172bcbda36" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 } health_report { overall_health: UNKNOWN } } }
I20260925 19:10:21.773969 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.771532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.27.174.65:37883)
0925 19:10:21.771590 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:10:21.771591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:10:21.771598 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:10:21.771669 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:10:21.771669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:10:21.771686 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:10:21.771761 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:10:21.771918 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.772366 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.773177 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:10:21.773380 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:10:21.773464 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:10:21.773634 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:10:21.773734 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:10:21.773825 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:10:21.773845 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 19:10:21.773864 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:10:21.775628 6499 raft_consensus.cc:1261] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 1 FOLLOWER]: Refusing update from remote peer fb9f30e8a07e4828b41bee172bcbda36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:10:21.775625 6365 raft_consensus.cc:1261] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 1 FOLLOWER]: Refusing update from remote peer fb9f30e8a07e4828b41bee172bcbda36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:10:21.775985 6569 consensus_queue.cc:1035] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24351300d33f4d22828222aa99632e2f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:10:21.776131 6564 consensus_queue.cc:1035] T 8f1bfd0e6ad34954b78a6d2575489e73 P fb9f30e8a07e4828b41bee172bcbda36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124efd19acb54d72b9a0da3d731af4bb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:10:21.778128 6572 mvcc.cc:205] Tried to move back new op lower bound from 7333328575590416384 to 7333328575543996416. Current Snapshot: MvccSnapshot[applied={T|T < 7333328575590416384}]
I20260925 19:10:21.778718 6574 mvcc.cc:205] Tried to move back new op lower bound from 7333328575590416384 to 7333328575543996416. Current Snapshot: MvccSnapshot[applied={T|T < 7333328575590416384}]
I20260925 19:10:21.778934 6573 mvcc.cc:205] Tried to move back new op lower bound from 7333328575590416384 to 7333328575543996416. Current Snapshot: MvccSnapshot[applied={T|T < 7333328575590416384}]
I20260925 19:10:21.780514 6086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:10:21.780615 6086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:10:21.781311 6086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3baf1458320f4c47876b21e9cc521a48: Sending DeleteTablet for 3 replicas of tablet 8f1bfd0e6ad34954b78a6d2575489e73
I20260925 19:10:21.781684 6211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:10:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56640
I20260925 19:10:21.781699 6479 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:10:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46052
I20260925 19:10:21.781913 6345 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1bfd0e6ad34954b78a6d2575489e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:10:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:58128
I20260925 19:10:21.782135 6583 tablet_replica.cc:330] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: stopping tablet replica
I20260925 19:10:21.782243 6583 raft_consensus.cc:2229] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:10:21.782342 6583 raft_consensus.cc:2258] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:10:21.782371 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 6148
I20260925 19:10:21.783212 6583 ts_tablet_manager.cc:1905] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:10:21.784121 6582 tablet_replica.cc:330] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: stopping tablet replica
I20260925 19:10:21.784224 6582 raft_consensus.cc:2229] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:10:21.784318 6582 raft_consensus.cc:2258] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:10:21.784737 6582 ts_tablet_manager.cc:1905] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:10:21.784942 6583 ts_tablet_manager.cc:1918] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:10:21.785019 6583 log.cc:1196] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/8f1bfd0e6ad34954b78a6d2575489e73
I20260925 19:10:21.785264 6583 ts_tablet_manager.cc:1939] T 8f1bfd0e6ad34954b78a6d2575489e73 P 124efd19acb54d72b9a0da3d731af4bb: Deleting consensus metadata
I20260925 19:10:21.785632 6071 catalog_manager.cc:4841] TS 124efd19acb54d72b9a0da3d731af4bb (127.27.174.66:40757): tablet 8f1bfd0e6ad34954b78a6d2575489e73 (table test-table [id=eca405be1cf54027b413a2b8b1f26217]) successfully deleted
I20260925 19:10:21.786464 6582 ts_tablet_manager.cc:1918] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:10:21.786550 6582 log.cc:1196] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/8f1bfd0e6ad34954b78a6d2575489e73
I20260925 19:10:21.786887 6582 ts_tablet_manager.cc:1939] T 8f1bfd0e6ad34954b78a6d2575489e73 P 24351300d33f4d22828222aa99632e2f: Deleting consensus metadata
I20260925 19:10:21.787369 6072 catalog_manager.cc:4841] TS 24351300d33f4d22828222aa99632e2f (127.27.174.67:35617): tablet 8f1bfd0e6ad34954b78a6d2575489e73 (table test-table [id=eca405be1cf54027b413a2b8b1f26217]) successfully deleted
I20260925 19:10:21.792140 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 6282
W20260925 19:10:21.792181 6073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:37883 (error 108)
I20260925 19:10:21.793608 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:10:21.793456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37883 (local address 127.0.0.1:56662)
0925 19:10:21.793516 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:10:21.793567 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:10:21.793748 6073 catalog_manager.cc:4584] TS fb9f30e8a07e4828b41bee172bcbda36 (127.27.174.65:37883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f1bfd0e6ad34954b78a6d2575489e73: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37883: connect: Connection refused (error 111)
I20260925 19:10:21.796922 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 6416
I20260925 19:10:21.801602 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 6054
2026-09-25T19:10:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1913 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6587](info): set up 2 sockets
Sep 25 19:10:21 dist-test-slave-fcdn krb5kdc[6587](info): commencing operation
krb5kdc: starting...
W20260925 19:10:22.832633 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:10:22 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:10:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
I20260925 19:10:22.996356 28345 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwy_z0T/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/postgres -l logfile start
2026-09-25 19:10:25.483 UTC [6617] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:10:25.483 UTC [6617] LOG: listening on IPv4 address "127.27.174.84", port 36839
2026-09-25 19:10:25.487 UTC [6617] LOG: listening on Unix socket "/tmp/.s.PGSQL.36839"
2026-09-25 19:10:25.521 UTC [6620] LOG: database system was shut down at 2026-09-25 19:10:24 UTC
2026-09-25 19:10:25.525 UTC [6617] LOG: database system is ready to accept connections
I20260925 19:10:26.476497 28345 mini_postgres.cc:96] Postgres bound to 36839
2026-09-25 19:10:26.482 UTC [6628] FATAL: database "slave" does not exist
127.27.174.84:36839 - accepting connections
I20260925 19:10:26.483103 28345 mini_ranger.cc:161] Starting Ranger...
I20260925 19:10:26.497414 28345 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:10:26.975234 28345 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:10:26.975320 28345 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin
2026-09-25 19:10:27,250 [I] DB FLAVOR :POSTGRES
2026-09-25 19:10:27,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:10:27,250 [I] Checking connection..
2026-09-25 19:10:27,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:10:27,516 [I] Checking connection passed.
2026-09-25 19:10:27,516 [I] --------- Verifying version history table ---------
2026-09-25 19:10:27,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:10:27,766 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:10:27,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:10:28,022 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:10:28,022 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:10:28,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:10:28.275 UTC [6692] WARNING: there is no transaction in progress
2026-09-25 19:10:28,281 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:10:28,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:10:28,505 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:10:28,505 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:10:28,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:10:28,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:10:28,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fcdn.c.gcp-upstream.internal','N') ;"
2026-09-25 19:10:29,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:10:29,476 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:10:29,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:10:29,727 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:10:29,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:10:29,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:10:30,248 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:10:30,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:10:31.207 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.225 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.242 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.260 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.277 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.298 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.356 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.364 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.380 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.391 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.402 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.416 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.426 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.432 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.440 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:31.447 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 19:10:32,109 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:10:32,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:10:32,330 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:10:32,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:10:32,584 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:10:32,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:10:32,827 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:10:32,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:10:33,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:10:33,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fcdn.c.gcp-upstream.internal';"
2026-09-25 19:10:33,568 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:10:33,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:10:33,806 [I] DB_PATCHES have already been applied
I20260925 19:10:33.812172 28345 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwy_z0T/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:10:33.812234 28345 mini_ranger.cc:193] Using host: 127.27.174.84
I20260925 19:10:33.815145 28345 mini_ranger.cc:227] Ranger admin URL: http://127.27.174.84:41469
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:10:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:10:34.914172 28345 mini_ranger.cc:160] Time spent starting Ranger: real 8.431s user 0.002s sys 0.006s
Sep 25, 2026 7:10:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:10:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 7:10:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:10:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:10:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:10:56.276780 28345 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:10:56 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363456, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:10:56.357815 28345 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:10:56.366838 28345 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:10:57.301517 28345 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:10:57.301605 28345 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms
I20260925 19:10:57.301625 28345 mini_ranger_kms.cc:227] Using postgres at 127.27.174.84:36839
2026-09-25 19:10:57,462 [I] DB FLAVOR :POSTGRES
2026-09-25 19:10:57,462 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:10:57,462 [I] Checking connection
2026-09-25 19:10:57,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:10:57,724 [I] connection success
2026-09-25 19:10:57,724 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:10:57,725 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:10:57,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:10:57,957 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:10:57,958 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:10:57,958 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:10:57,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:36839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:10:58,216 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:10:58.696424 28345 mini_ranger_kms.cc:310] Created kms service
I20260925 19:10:58.896999 28345 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:10:58.941529 28345 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:10:58 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363456, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:10:59.321136 28345 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:10:59.321251 28345 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:10:59.321273 28345 mini_ranger_kms.cc:240] Using host: 127.27.174.84
I20260925 19:10:59.324518 28345 mini_ranger_kms.cc:276] Ranger KMS PID: 6980
I20260925 19:10:59.324648 28345 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.174.84:33727
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:11:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 19:11:01 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:01.587073 28345 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.229s user 0.002s sys 0.142s
I20260925 19:11:01.587188 28345 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:11:01.587276 28345 mini_ranger_kms.cc:401] 127.27.174.84:33727/kms/v1/keys
Sep 25 19:11:01 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
Sep 25 19:11:02 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
Sep 25 19:11:04 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:11:04.298859 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38165
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38165
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:33727/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:04.396852 7031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:04.397047 7031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:04.397068 7031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:04.398205 7031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:04.398252 7031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:04.398265 7031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:04.398278 7031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:04.398290 7031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:04.399844 7031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.174.84:33727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38165
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:04.400077 7031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:04.400309 7031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:04.406343 7036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:04.406558 7031 server_base.cc:1034] running on GCE node
W20260925 19:11:04.406975 7039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:04.406955 7037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:04.407310 7031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:04.407701 7031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:04.408892 7031 hybrid_clock.cc:648] HybridClock initialized: now 1790363464408854 us; error 47 us; skew 500 ppm
Sep 25 19:11:04 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363464, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:04.416838 7031 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:11:04.417393 7031 webserver.cc:460] Webserver started at http://127.27.174.126:34221/ using document root <none> and password file <none>
I20260925 19:11:04.417637 7031 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:04.417695 7031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:04.417826 7031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:04 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363464, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:04.475281 7031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "6075876d19e1415980e644f717731f81"
format_stamp: "Formatted at 2026-09-25 19:11:04 on dist-test-slave-fcdn"
server_key: "7c066de67a66845d79400190bea17fc0"
server_key_iv: "61f63df90429e25de92f1a57e47b90fd"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:04.475775 7031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "6075876d19e1415980e644f717731f81"
format_stamp: "Formatted at 2026-09-25 19:11:04 on dist-test-slave-fcdn"
server_key: "7c066de67a66845d79400190bea17fc0"
server_key_iv: "61f63df90429e25de92f1a57e47b90fd"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:04.477648 7031 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 19:11:04.489135 7046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:04.489372 7031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:04.489461 7031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "6075876d19e1415980e644f717731f81"
format_stamp: "Formatted at 2026-09-25 19:11:04 on dist-test-slave-fcdn"
server_key: "7c066de67a66845d79400190bea17fc0"
server_key_iv: "61f63df90429e25de92f1a57e47b90fd"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:04.489560 7031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:04.505342 7031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:04.507373 7031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:04.507536 7031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:04.512025 7031 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38165
I20260925 19:11:04.512086 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38165 every 8 connection(s)
I20260925 19:11:04.512445 7031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:11:04.513105 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:04.515399 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 7031
I20260925 19:11:04.515511 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:11:04.515743 28345 external_mini_cluster.cc:1428] Setting key 562c47cc504cae77536a2bba948b55ea
I20260925 19:11:04.516131 7099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: Bootstrap starting.
I20260925 19:11:04.516880 7099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:04.517195 7099 log.cc:824] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:04.517905 7099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: No bootstrap required, opened a new log
I20260925 19:11:04.520222 7099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } }
I20260925 19:11:04.520382 7099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:04.520423 7099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6075876d19e1415980e644f717731f81, State: Initialized, Role: FOLLOWER
I20260925 19:11:04.520545 7099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } }
I20260925 19:11:04.520625 7099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:11:04.520664 7099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:11:04.520702 7099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:04.521335 7099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } }
I20260925 19:11:04.521432 7099 leader_election.cc:302] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6075876d19e1415980e644f717731f81; no voters:
I20260925 19:11:04.521574 7099 leader_election.cc:288] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:11:04.521793 7105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:04.521831 7099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:04.521993 7105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [term 1 LEADER]: Becoming Leader. State: Replica: 6075876d19e1415980e644f717731f81, State: Running, Role: LEADER
I20260925 19:11:04.522135 7105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } }
I20260925 19:11:04.522498 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6075876d19e1415980e644f717731f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } } }
I20260925 19:11:04.522667 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:04.522830 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6075876d19e1415980e644f717731f81. Latest consensus state: current_term: 1 leader_uuid: "6075876d19e1415980e644f717731f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6075876d19e1415980e644f717731f81" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38165 } } }
I20260925 19:11:04.522948 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:04.523142 7113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:11:04.523344 7099 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:11:04.523708 7113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:11:04 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:04.527887 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:04.516378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.27.174.126:38165)
0925 19:11:04.516545 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:04.516556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:04.516587 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:11:04.517201 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:04.517210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:04.517272 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:04.517427 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:04.518169 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:04.519635 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:04.520728 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:04.521351 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:04.521509 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:04.524288 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:04.524292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:04.524293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:04.524304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:04.526277 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:04.526622 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:04.526625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:04.526626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:04.526720 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:04.526921 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:04.526922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:04.526922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:04.527236 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:04.527381 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:04.527523 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:11:04.527680 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260925 19:11:04.531504 7113 catalog_manager.cc:1347] Generated new cluster ID: 4f5c1f71d09442789b516404c1abb70b
I20260925 19:11:04.531635 7113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:11:04.544569 7113 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:11:04.545349 7113 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:11:04.551319 7113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: Generated new TSK 0
I20260925 19:11:04.551503 7113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:11:05.258493 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:05.345359 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:05.345594 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:05.345633 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:05.346846 7144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:05.346918 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:05.346962 7144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:11:05.348554 7144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.174.84:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:05.348835 7144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:05.349124 7144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:05.354357 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.354357 7152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.354357 7150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:05.354619 7144 server_base.cc:1034] running on GCE node
I20260925 19:11:05.354938 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:05.355237 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:05.356426 7144 hybrid_clock.cc:648] HybridClock initialized: now 1790363465356410 us; error 33 us; skew 500 ppm
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:05.359299 7144 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:11:05.359858 7144 webserver.cc:460] Webserver started at http://127.27.174.65:38485/ using document root <none> and password file <none>
I20260925 19:11:05.360150 7144 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:05.360251 7144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:05.360440 7144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:05.374055 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "82f08237d10449b39a7aba68dde0759a"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "e22539b8d90b97512362b74fed8de927"
server_key_iv: "f7b0cd728dec13aaaa64334de8f63a71"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.374629 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "82f08237d10449b39a7aba68dde0759a"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "e22539b8d90b97512362b74fed8de927"
server_key_iv: "f7b0cd728dec13aaaa64334de8f63a71"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.383299 7144 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 19:11:05.399778 7159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.399986 7144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 19:11:05.400103 7144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "82f08237d10449b39a7aba68dde0759a"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "e22539b8d90b97512362b74fed8de927"
server_key_iv: "f7b0cd728dec13aaaa64334de8f63a71"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.400233 7144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:05.416744 7144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:05.419209 7144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:05.419412 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:05.419723 7144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:05.420127 7144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:05.420219 7144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.420322 7144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:05.420399 7144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.427279 7144 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35465
I20260925 19:11:05.427851 7144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:11:05.434226 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 7144
I20260925 19:11:05.434343 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:11:05.434556 28345 external_mini_cluster.cc:1428] Setting key c80f1392f321bd7b09489d65c7a7c30d
I20260925 19:11:05.437400 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35465 every 8 connection(s)
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20260925 19:11:05.456724 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.429046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38165 (local address 127.27.174.65:34643)
0925 19:11:05.437713 (+ 8667us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:05.437764 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:05.439762 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:05.440170 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:05.440176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:05.440440 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:05.440750 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.440756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.441788 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.441789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.443084 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.443087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.443685 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.443686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.443835 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.444503 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:05.444514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:05.446321 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:05.450286 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.450297 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.450300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.450557 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.450806 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.450807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.450808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.450948 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.455915 (+ 4967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:05.455927 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:05.456135 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 19:11:05.456266 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 19:11:05.456446 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8605,"thread_start_us":84,"threads_started":1}
I20260925 19:11:05.457150 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.429097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34643 (local address 127.27.174.126:38165)
0925 19:11:05.439583 (+ 10486us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:05.439586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:05.439625 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:11:05.439940 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:05.439941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:05.439956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:05.440025 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:05.440868 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.441660 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.443211 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.443511 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.443556 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.446440 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:05.446442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:05.446442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:05.446450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:05.450162 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.450652 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.450654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.450654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.450728 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.451023 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.451024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.451025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.455785 (+ 4760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:05.455879 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:05.456949 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0925 19:11:05.457045 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10453,"thread_start_us":105,"threads_started":1}
I20260925 19:11:05.457837 7275 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38165
I20260925 19:11:05.457937 7275 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:05.458129 7275 heartbeater.cc:511] Master 127.27.174.126:38165 requested a full tablet report, sending...
I20260925 19:11:05.458706 7063 ts_manager.cc:194] Registered new tserver with Master: 82f08237d10449b39a7aba68dde0759a (127.27.174.65:35465)
I20260925 19:11:05.459687 7063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:34643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:11:05.478538 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 19:11:05.600061 7281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:05.600404 7281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:05.600483 7281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:05.602242 7281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:05.602300 7281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:05.602341 7281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:11:05.604688 7281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.174.84:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:05.605085 7281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:05.605374 7281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:05.611941 7289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.611951 7291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.612866 7288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:05.613128 7281 server_base.cc:1034] running on GCE node
I20260925 19:11:05.613426 7281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:05.613686 7281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:05.617012 7281 hybrid_clock.cc:648] HybridClock initialized: now 1790363465616976 us; error 48 us; skew 500 ppm
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:05.620250 7281 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:11:05.620766 7281 webserver.cc:460] Webserver started at http://127.27.174.66:41785/ using document root <none> and password file <none>
I20260925 19:11:05.621014 7281 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:05.621075 7281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:05.621213 7281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:05.639642 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "97170c50afad4a5281b4f8fc2dd606cd"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "c3562736961ee78140d867129dad1e16"
server_key_iv: "0b7e5bfe5e32831e3296a12126fb3ec1"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.640127 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "97170c50afad4a5281b4f8fc2dd606cd"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "c3562736961ee78140d867129dad1e16"
server_key_iv: "0b7e5bfe5e32831e3296a12126fb3ec1"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.648380 7281 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.002s
I20260925 19:11:05.664877 7298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.671913 7281 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260925 19:11:05.672027 7281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "97170c50afad4a5281b4f8fc2dd606cd"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "c3562736961ee78140d867129dad1e16"
server_key_iv: "0b7e5bfe5e32831e3296a12126fb3ec1"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.672132 7281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:05.697151 7281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:05.699777 7281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:05.699971 7281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:05.700219 7281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:05.700592 7281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:05.700688 7281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.700760 7281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:05.700815 7281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.706988 7281 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:43129
I20260925 19:11:05.707459 7281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:11:05.708832 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 7281
I20260925 19:11:05.708946 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:11:05.709149 28345 external_mini_cluster.cc:1428] Setting key e97c0d1cbc34cdab6af24d38b787343c
I20260925 19:11:05.709681 7412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:43129 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:05.730165 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.712213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45969 (local address 127.27.174.126:38165)
0925 19:11:05.712275 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:05.712278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:05.712732 (+ 454us) server_negotiation.cc:407] Connection header received
0925 19:11:05.713390 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:05.713391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:05.713407 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:05.713478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:05.714307 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.715051 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.723651 (+ 8600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.723944 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.723996 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.726779 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:05.726780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:05.726780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:05.726789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:05.728529 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.729035 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.729036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.729037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.729096 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.729396 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.729397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.729397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.729537 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:05.729578 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:05.729968 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 19:11:05.730062 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:11:05.731191 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.712208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38165 (local address 127.27.174.66:45969)
0925 19:11:05.712576 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:05.712617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:05.713267 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:05.713616 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:05.713620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:05.713880 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:05.714187 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.714195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.715182 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.715183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.723489 (+ 8306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.723493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.724106 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.724107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.724262 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.724936 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:05.724947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:05.726636 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:05.728629 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.728635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.728638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.728940 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.729169 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.729170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.729170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.729316 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.729622 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:05.729627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:05.729808 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 19:11:05.730769 (+ 961us) client_negotiation.cc:240] Negotiation successful
0925 19:11:05.730950 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":85,"threads_started":1}
I20260925 19:11:05.731904 7413 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38165
I20260925 19:11:05.731998 7413 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:05.732187 7413 heartbeater.cc:511] Master 127.27.174.126:38165 requested a full tablet report, sending...
I20260925 19:11:05.732631 7063 ts_manager.cc:194] Registered new tserver with Master: 97170c50afad4a5281b4f8fc2dd606cd (127.27.174.66:43129)
I20260925 19:11:05.733018 7063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:45969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:11:05.746613 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:33727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:05.865691 7418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:05.866050 7418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:05.866142 7418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:05.867947 7418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:05.868036 7418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:05.868098 7418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:11:05.870352 7418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.174.84:33727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:05.870752 7418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:05.871049 7418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:05.877239 7426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.877360 7423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:05.877537 7424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:05.878753 7418 server_base.cc:1034] running on GCE node
I20260925 19:11:05.878970 7418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:05.879233 7418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:05.880369 7418 hybrid_clock.cc:648] HybridClock initialized: now 1790363465880346 us; error 35 us; skew 500 ppm
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:05.882843 7418 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:11:05.883275 7418 webserver.cc:460] Webserver started at http://127.27.174.67:44237/ using document root <none> and password file <none>
I20260925 19:11:05.883517 7418 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:05.883585 7418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:05.883749 7418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:05.897143 7418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "2d551f40d94541539d0885dda1931747"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "a085154fd596361a656ced8b5da21b58"
server_key_iv: "10ded3dfafdd3332b160a2818c3a3502"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.897604 7418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "2d551f40d94541539d0885dda1931747"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "a085154fd596361a656ced8b5da21b58"
server_key_iv: "10ded3dfafdd3332b160a2818c3a3502"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.900517 7418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 19:11:05.937359 7433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.937652 7418 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260925 19:11:05.937759 7418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2d551f40d94541539d0885dda1931747"
format_stamp: "Formatted at 2026-09-25 19:11:05 on dist-test-slave-fcdn"
server_key: "a085154fd596361a656ced8b5da21b58"
server_key_iv: "10ded3dfafdd3332b160a2818c3a3502"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:05.937865 7418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:05.953871 7418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:05.956207 7418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:05.956382 7418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:05.956655 7418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:05.957032 7418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:05.957088 7418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.957161 7418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:05.957190 7418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:05.964262 7418 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37169
I20260925 19:11:05.964696 7418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:11:05.966084 7546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37169 every 8 connection(s)
Sep 25 19:11:05 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:05.974586 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 7418
I20260925 19:11:05.974709 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:11:05.974931 28345 external_mini_cluster.cc:1428] Setting key 8aaf3f65ffbc1c304f46c7a177883172
I20260925 19:11:05.976253 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.965782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:46255 (local address 127.27.174.126:38165)
0925 19:11:05.965832 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:05.965834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:05.966694 (+ 860us) server_negotiation.cc:407] Connection header received
0925 19:11:05.967248 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:05.967250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:05.967263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:05.967313 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:05.968015 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.968525 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.969540 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.969742 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.969827 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.972857 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:05.972859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:05.972860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:05.972867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:05.974292 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.974823 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.974825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.974825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.974884 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.975455 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.975456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.975456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.975595 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:05.975674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:05.976078 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 19:11:05.976160 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:11:05.976567 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.965464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38165 (local address 127.27.174.67:46255)
0925 19:11:05.966555 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:05.966587 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:05.967123 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:05.967434 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:05.967437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:05.967705 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:05.967914 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.967919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.968636 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.968637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.969440 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:05.969442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.969836 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.969838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:05.969967 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.971311 (+ 1344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:05.971320 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:05.972757 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:05.974393 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.974400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.974402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.974722 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.974959 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:05.974960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:05.974960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:05.975375 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:05.975725 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:05.975732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:05.975930 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 19:11:05.976043 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 19:11:05.976197 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":82,"threads_started":1}
I20260925 19:11:05.977020 7547 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38165
I20260925 19:11:05.977139 7547 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:05.977332 7547 heartbeater.cc:511] Master 127.27.174.126:38165 requested a full tablet report, sending...
I20260925 19:11:05.977807 7063 ts_manager.cc:194] Registered new tserver with Master: 2d551f40d94541539d0885dda1931747 (127.27.174.67:37169)
I20260925 19:11:05.978191 7063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:46255
I20260925 19:11:05.987416 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:11:05.997216 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:05.989114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.27.174.126:38165)
0925 19:11:05.989161 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:05.989163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:05.989321 (+ 158us) server_negotiation.cc:407] Connection header received
0925 19:11:05.989391 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:05.989392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:05.989402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:05.989458 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:05.990042 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.990638 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.991661 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:05.991953 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:05.991999 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:05.993540 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:05.993541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:05.993542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:05.993550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:05.995889 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.996312 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.996314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.996314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.996373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:05.996590 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:05.996590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:05.996591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:05.996738 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:05.996815 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:05.997040 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 19:11:05.997121 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:11:05.999490 7063 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:11:06.001588 7063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:11:06.009287 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.004624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37169 (local address 127.0.0.1:38204)
0925 19:11:06.005454 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.005465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:06.005518 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:06.005722 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:06.005725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:06.005727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:06.005792 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.005794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.006639 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.006640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.007878 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.007881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.008969 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.008970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.009056 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.009072 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:11:06.009139 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:11:06.009201 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":87,"threads_started":1}
I20260925 19:11:06.012687 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.005037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.27.174.67:37169)
0925 19:11:06.005252 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.005257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.005514 (+ 257us) server_negotiation.cc:407] Connection header received
0925 19:11:06.005586 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.005588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.005605 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.005701 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:06.005848 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.006536 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.008008 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.008776 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.008848 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.008929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.012555 (+ 3626us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.012605 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260925 19:11:06.013342 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 101aad36602a4aeeb857beafff04c1da (DEFAULT_TABLE table=test-table [id=72214dd25096475a89337e586e32cd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:06.013684 7481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101aad36602a4aeeb857beafff04c1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:06.018524 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.004395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35465 (local address 127.0.0.1:45272)
0925 19:11:06.004528 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.004573 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:06.004634 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:06.014485 (+ 9851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:06.014486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:06.014488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:06.014530 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.014531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.015806 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.015807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.017070 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.017073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.018283 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.018284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.018351 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.018354 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:11:06.018385 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 19:11:06.018433 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260925 19:11:06.018604 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.005058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.27.174.66:43129)
0925 19:11:06.005403 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.005407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.009865 (+ 4458us) server_negotiation.cc:407] Connection header received
0925 19:11:06.009988 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.009991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.010010 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.010067 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:06.010283 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.011316 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.017361 (+ 6045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.018115 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.018166 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.018240 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.018476 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.018517 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":82,"threads_started":1}
I20260925 19:11:06.018860 7565 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Bootstrap starting.
I20260925 19:11:06.019232 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.004485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.27.174.65:35465)
0925 19:11:06.014293 (+ 9808us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.014298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.014312 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:11:06.014352 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.014354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.014371 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.014475 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:06.014621 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.015700 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.017189 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.018182 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.018994 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.019057 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.019082 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.019130 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9769,"thread_start_us":83,"threads_started":1}
I20260925 19:11:06.019739 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.004836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:43129 (local address 127.0.0.1:33054)
0925 19:11:06.009811 (+ 4975us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.009820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:06.009893 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:06.010172 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:06.010173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:06.010174 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:06.010205 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.010206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.011412 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.011412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.012449 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.012452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.018291 (+ 5839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.018291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.018356 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.018358 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:11:06.018408 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:11:06.018438 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4963,"mutex_wait_us":12,"thread_start_us":63,"threads_started":1}
I20260925 19:11:06.019968 7565 tablet_bootstrap.cc:654] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:06.020073 7209 tablet_service.cc:1467] Processing CreateTablet for tablet 101aad36602a4aeeb857beafff04c1da (DEFAULT_TABLE table=test-table [id=72214dd25096475a89337e586e32cd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:06.020330 7565 log.cc:824] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:06.020428 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101aad36602a4aeeb857beafff04c1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:06.020602 7336 tablet_service.cc:1467] Processing CreateTablet for tablet 101aad36602a4aeeb857beafff04c1da (DEFAULT_TABLE table=test-table [id=72214dd25096475a89337e586e32cd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:06.020897 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101aad36602a4aeeb857beafff04c1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:06.021082 7565 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: No bootstrap required, opened a new log
I20260925 19:11:06.021136 7565 ts_tablet_manager.cc:1397] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:11:06.023066 7565 raft_consensus.cc:348] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.023231 7565 raft_consensus.cc:374] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:06.023264 7565 raft_consensus.cc:729] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d551f40d94541539d0885dda1931747, State: Initialized, Role: FOLLOWER
I20260925 19:11:06.023370 7565 consensus_queue.cc:260] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.023654 7565 ts_tablet_manager.cc:1428] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:11:06.023811 7547 heartbeater.cc:503] Master 127.27.174.126:38165 was elected leader, sending a full tablet report...
I20260925 19:11:06.024770 7569 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Bootstrap starting.
I20260925 19:11:06.025156 7567 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Bootstrap starting.
I20260925 19:11:06.025830 7569 tablet_bootstrap.cc:654] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:06.025871 7567 tablet_bootstrap.cc:654] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:06.026201 7567 log.cc:824] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:06.026206 7569 log.cc:824] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:06.026975 7569 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: No bootstrap required, opened a new log
I20260925 19:11:06.027041 7569 ts_tablet_manager.cc:1397] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:11:06.027128 7567 tablet_bootstrap.cc:492] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: No bootstrap required, opened a new log
I20260925 19:11:06.027187 7567 ts_tablet_manager.cc:1397] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:11:06.029086 7569 raft_consensus.cc:348] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.029258 7569 raft_consensus.cc:374] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:06.029291 7569 raft_consensus.cc:729] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97170c50afad4a5281b4f8fc2dd606cd, State: Initialized, Role: FOLLOWER
I20260925 19:11:06.029403 7569 consensus_queue.cc:260] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.029675 7569 ts_tablet_manager.cc:1428] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:11:06.029933 7413 heartbeater.cc:503] Master 127.27.174.126:38165 was elected leader, sending a full tablet report...
I20260925 19:11:06.030079 7567 raft_consensus.cc:348] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.030231 7567 raft_consensus.cc:374] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:06.030277 7567 raft_consensus.cc:729] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f08237d10449b39a7aba68dde0759a, State: Initialized, Role: FOLLOWER
I20260925 19:11:06.030401 7567 consensus_queue.cc:260] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.030678 7567 ts_tablet_manager.cc:1428] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:11:06.030826 7275 heartbeater.cc:503] Master 127.27.174.126:38165 was elected leader, sending a full tablet report...
Sep 25 19:11:06 dist-test-slave-fcdn krb5kdc[6587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:06.150964 7573 raft_consensus.cc:482] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:11:06.151153 7573 raft_consensus.cc:504] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.151532 7573 leader_election.cc:288] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97170c50afad4a5281b4f8fc2dd606cd (127.27.174.66:43129), 2d551f40d94541539d0885dda1931747 (127.27.174.67:37169)
I20260925 19:11:06.156713 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.152024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43781 (local address 127.27.174.67:37169)
0925 19:11:06.152070 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.152072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.153171 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 19:11:06.153278 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.153279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.153291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.153319 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:06.153544 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.154285 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.155766 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.156326 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.156377 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.156387 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.156599 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.156640 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:11:06.156773 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.152716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42143 (local address 127.27.174.66:43129)
0925 19:11:06.152756 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.152758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.153671 (+ 913us) server_negotiation.cc:407] Connection header received
0925 19:11:06.153773 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.153774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.153784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.153813 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:06.154042 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.154664 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.155929 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.156412 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.156458 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.156465 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.156670 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.156708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:11:06.156992 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.152498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:43129 (local address 127.27.174.65:42143)
0925 19:11:06.153586 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.153599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:06.153699 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:06.153875 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:06.153876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:06.153878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:06.153946 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.153947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.154782 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.154783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.155831 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.155833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.156522 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.156522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.156594 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.156597 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:11:06.156885 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 19:11:06.156922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":98,"threads_started":1}
I20260925 19:11:06.157298 7365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101aad36602a4aeeb857beafff04c1da" candidate_uuid: "82f08237d10449b39a7aba68dde0759a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97170c50afad4a5281b4f8fc2dd606cd" is_pre_election: true
I20260925 19:11:06.157371 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.151809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37169 (local address 127.27.174.65:43781)
0925 19:11:06.153095 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.153106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:06.153201 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:06.153386 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:06.153387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:06.153389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:06.153473 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.153475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.154408 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.154408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.155666 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:06.155669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.156448 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.156449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:06.156519 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.156522 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:11:06.157230 (+ 708us) client_negotiation.cc:240] Negotiation successful
0925 19:11:06.157273 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"thread_start_us":108,"threads_started":1}
I20260925 19:11:06.157485 7365 raft_consensus.cc:2454] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f08237d10449b39a7aba68dde0759a in term 0.
I20260925 19:11:06.157734 7163 leader_election.cc:302] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f08237d10449b39a7aba68dde0759a, 97170c50afad4a5281b4f8fc2dd606cd; no voters:
I20260925 19:11:06.157866 7573 raft_consensus.cc:2781] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:11:06.157908 7573 raft_consensus.cc:482] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:11:06.157860 7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101aad36602a4aeeb857beafff04c1da" candidate_uuid: "82f08237d10449b39a7aba68dde0759a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d551f40d94541539d0885dda1931747" is_pre_election: true
I20260925 19:11:06.157930 7573 raft_consensus.cc:3037] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:06.157989 7486 raft_consensus.cc:2454] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f08237d10449b39a7aba68dde0759a in term 0.
I20260925 19:11:06.158845 7573 raft_consensus.cc:504] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.158980 7573 leader_election.cc:288] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [CANDIDATE]: Term 1 election: Requested vote from peers 97170c50afad4a5281b4f8fc2dd606cd (127.27.174.66:43129), 2d551f40d94541539d0885dda1931747 (127.27.174.67:37169)
I20260925 19:11:06.159162 7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101aad36602a4aeeb857beafff04c1da" candidate_uuid: "82f08237d10449b39a7aba68dde0759a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d551f40d94541539d0885dda1931747"
I20260925 19:11:06.159240 7486 raft_consensus.cc:3037] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:06.159802 7365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101aad36602a4aeeb857beafff04c1da" candidate_uuid: "82f08237d10449b39a7aba68dde0759a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97170c50afad4a5281b4f8fc2dd606cd"
I20260925 19:11:06.159891 7365 raft_consensus.cc:3037] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:06.160795 7365 raft_consensus.cc:2454] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f08237d10449b39a7aba68dde0759a in term 1.
I20260925 19:11:06.160993 7163 leader_election.cc:302] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f08237d10449b39a7aba68dde0759a, 97170c50afad4a5281b4f8fc2dd606cd; no voters:
I20260925 19:11:06.161607 7486 raft_consensus.cc:2454] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f08237d10449b39a7aba68dde0759a in term 1.
I20260925 19:11:06.161811 7573 raft_consensus.cc:2781] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:06.161966 7573 raft_consensus.cc:686] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [term 1 LEADER]: Becoming Leader. State: Replica: 82f08237d10449b39a7aba68dde0759a, State: Running, Role: LEADER
I20260925 19:11:06.162074 7573 consensus_queue.cc:237] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } }
I20260925 19:11:06.162904 7063 catalog_manager.cc:5495] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f08237d10449b39a7aba68dde0759a (127.27.174.65). New cstate: current_term: 1 leader_uuid: "82f08237d10449b39a7aba68dde0759a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f08237d10449b39a7aba68dde0759a" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 } health_report { overall_health: UNKNOWN } } }
W20260925 19:11:06.179000 7276 tablet.cc:2369] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:11:06.194341 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.190913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.27.174.65:35465)
0925 19:11:06.190954 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:06.190956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:06.190965 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:11:06.191007 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:06.191008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:06.191029 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:06.191117 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:11:06.191329 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.192102 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.193187 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:06.193484 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:06.193538 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:06.193755 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:11:06.193886 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:11:06.193982 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:06.194219 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 19:11:06.194241 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:11:06.196298 7486 raft_consensus.cc:1261] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 1 FOLLOWER]: Refusing update from remote peer 82f08237d10449b39a7aba68dde0759a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:06.196617 7573 consensus_queue.cc:1035] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d551f40d94541539d0885dda1931747" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:06.198341 7365 raft_consensus.cc:1261] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 1 FOLLOWER]: Refusing update from remote peer 82f08237d10449b39a7aba68dde0759a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:06.198588 7573 consensus_queue.cc:1035] T 101aad36602a4aeeb857beafff04c1da P 82f08237d10449b39a7aba68dde0759a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97170c50afad4a5281b4f8fc2dd606cd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:06.206327 7579 mvcc.cc:205] Tried to move back new op lower bound from 7333328757537263616 to 7333328757400571904. Current Snapshot: MvccSnapshot[applied={T|T < 7333328757537263616}]
I20260925 19:11:06.206487 7580 mvcc.cc:205] Tried to move back new op lower bound from 7333328757537263616 to 7333328757400571904. Current Snapshot: MvccSnapshot[applied={T|T < 7333328757537263616}]
I20260925 19:11:06.206663 7582 mvcc.cc:205] Tried to move back new op lower bound from 7333328757537263616 to 7333328757400571904. Current Snapshot: MvccSnapshot[applied={T|T < 7333328757537263616}]
I20260925 19:11:06.210255 7063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:11:06.210357 7063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:11:06.211024 7063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6075876d19e1415980e644f717731f81: Sending DeleteTablet for 3 replicas of tablet 101aad36602a4aeeb857beafff04c1da
I20260925 19:11:06.211521 7209 tablet_service.cc:1514] Processing DeleteTablet for tablet 101aad36602a4aeeb857beafff04c1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260925 19:11:06.211762 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 101aad36602a4aeeb857beafff04c1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:38204
I20260925 19:11:06.211956 7336 tablet_service.cc:1514] Processing DeleteTablet for tablet 101aad36602a4aeeb857beafff04c1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33054
I20260925 19:11:06.212493 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 7144
I20260925 19:11:06.214553 7590 tablet_replica.cc:330] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: stopping tablet replica
I20260925 19:11:06.214663 7590 raft_consensus.cc:2229] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:11:06.214773 7590 raft_consensus.cc:2258] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:11:06.214929 7591 tablet_replica.cc:330] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: stopping tablet replica
I20260925 19:11:06.215018 7591 raft_consensus.cc:2229] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:11:06.215119 7591 raft_consensus.cc:2258] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:11:06.215549 7590 ts_tablet_manager.cc:1905] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:11:06.216801 7591 ts_tablet_manager.cc:1905] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:11:06.217375 7590 ts_tablet_manager.cc:1918] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:11:06.217453 7590 log.cc:1196] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/101aad36602a4aeeb857beafff04c1da
I20260925 19:11:06.218127 7590 ts_tablet_manager.cc:1939] T 101aad36602a4aeeb857beafff04c1da P 2d551f40d94541539d0885dda1931747: Deleting consensus metadata
I20260925 19:11:06.218523 7049 catalog_manager.cc:4841] TS 2d551f40d94541539d0885dda1931747 (127.27.174.67:37169): tablet 101aad36602a4aeeb857beafff04c1da (table test-table [id=72214dd25096475a89337e586e32cd68]) successfully deleted
I20260925 19:11:06.221570 7591 ts_tablet_manager.cc:1918] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:11:06.221701 7591 log.cc:1196] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/101aad36602a4aeeb857beafff04c1da
I20260925 19:11:06.222007 7591 ts_tablet_manager.cc:1939] T 101aad36602a4aeeb857beafff04c1da P 97170c50afad4a5281b4f8fc2dd606cd: Deleting consensus metadata
I20260925 19:11:06.222446 7050 catalog_manager.cc:4841] TS 97170c50afad4a5281b4f8fc2dd606cd (127.27.174.66:43129): tablet 101aad36602a4aeeb857beafff04c1da (table test-table [id=72214dd25096475a89337e586e32cd68]) successfully deleted
W20260925 19:11:06.222844 7049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:35465 (error 108)
I20260925 19:11:06.223781 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 7281
I20260925 19:11:06.223802 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:06.223663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35465 (local address 127.0.0.1:45290)
0925 19:11:06.223710 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:06.223764 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 19:11:06.223922 7049 catalog_manager.cc:4584] TS 82f08237d10449b39a7aba68dde0759a (127.27.174.65:35465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 101aad36602a4aeeb857beafff04c1da: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35465: connect: Connection refused (error 111)
I20260925 19:11:06.230154 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 7418
I20260925 19:11:06.235464 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 7031
I20260925 19:11:06.245797 28345 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:11:06.580469 28345 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:11:06.580574 28345 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:11:06.924516 28345 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:11:06.924 UTC [6617] LOG: received smart shutdown request
2026-09-25 19:11:06.928 UTC [6617] LOG: background worker "logical replication launcher" (PID 6626) exited with exit code 1
2026-09-25 19:11:06.928 UTC [6621] LOG: shutting down
2026-09-25 19:11:07.092 UTC [6617] LOG: database system is shut down
2026-09-25T19:11:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45400 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:11:07 dist-test-slave-fcdn krb5kdc[7601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:11:07 dist-test-slave-fcdn krb5kdc[7601](info): set up 2 sockets
Sep 25 19:11:07 dist-test-slave-fcdn krb5kdc[7601](info): commencing operation
krb5kdc: starting...
W20260925 19:11:08.247568 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:11:08 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:11:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
I20260925 19:11:08.406307 28345 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwy_z0T/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/postgres -l logfile start
2026-09-25 19:11:10.872 UTC [7631] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:11:10.872 UTC [7631] LOG: listening on IPv4 address "127.27.174.84", port 32953
2026-09-25 19:11:10.875 UTC [7631] LOG: listening on Unix socket "/tmp/.s.PGSQL.32953"
2026-09-25 19:11:10.895 UTC [7634] LOG: database system was shut down at 2026-09-25 19:11:09 UTC
2026-09-25 19:11:10.899 UTC [7631] LOG: database system is ready to accept connections
I20260925 19:11:11.855335 28345 mini_postgres.cc:96] Postgres bound to 32953
2026-09-25 19:11:11.861 UTC [7642] FATAL: database "slave" does not exist
127.27.174.84:32953 - accepting connections
I20260925 19:11:11.861610 28345 mini_ranger.cc:161] Starting Ranger...
I20260925 19:11:11.876000 28345 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:11:12.373261 28345 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:11:12.373361 28345 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin
2026-09-25 19:11:12,646 [I] DB FLAVOR :POSTGRES
2026-09-25 19:11:12,646 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:11:12,646 [I] Checking connection..
2026-09-25 19:11:12,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:11:12,877 [I] Checking connection passed.
2026-09-25 19:11:12,878 [I] --------- Verifying version history table ---------
2026-09-25 19:11:12,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:11:13,140 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:11:13,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:11:13,391 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:11:13,391 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:11:13,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:11:13.644 UTC [7706] WARNING: there is no transaction in progress
2026-09-25 19:11:13,650 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:11:13,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:11:13,906 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:11:13,906 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:11:13,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:11:14,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:11:14,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fcdn.c.gcp-upstream.internal','N') ;"
2026-09-25 19:11:14,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:11:14,867 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:11:14,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:11:15,103 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:11:15,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:11:15,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:11:15,581 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:11:15,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:11:16.573 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.589 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.606 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.621 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.639 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.661 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.718 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.724 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.739 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.749 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.759 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.771 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.781 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.787 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.794 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:16.800 UTC [7814] WARNING: there is no transaction in progress
2026-09-25 19:11:17,451 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:11:17,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:11:17,710 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:11:17,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:11:17,945 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:11:17,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:11:18,190 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:11:18,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:11:18,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:11:18,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fcdn.c.gcp-upstream.internal';"
2026-09-25 19:11:18,888 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:11:18,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:11:19,138 [I] DB_PATCHES have already been applied
I20260925 19:11:19.144192 28345 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwy_z0T/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:11:19.144277 28345 mini_ranger.cc:193] Using host: 127.27.174.84
I20260925 19:11:19.147261 28345 mini_ranger.cc:227] Ranger admin URL: http://127.27.174.84:53875
Sep 25, 2026 7:11:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:11:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:11:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:11:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:11:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:11:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:11:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:11:20.262984 28345 mini_ranger.cc:160] Time spent starting Ranger: real 8.401s user 0.000s sys 0.008s
Sep 25, 2026 7:11:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:11:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 7:11:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:11:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:11:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:11:42.143956 28345 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:11:42 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:11:42.238116 28345 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:11:42.247044 28345 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:11:43.149191 28345 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:11:43.149282 28345 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms
I20260925 19:11:43.149305 28345 mini_ranger_kms.cc:227] Using postgres at 127.27.174.84:32953
2026-09-25 19:11:43,318 [I] DB FLAVOR :POSTGRES
2026-09-25 19:11:43,319 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:11:43,319 [I] Checking connection
2026-09-25 19:11:43,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:11:43,551 [I] connection success
2026-09-25 19:11:43,551 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:11:43,552 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:11:43,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:11:43,779 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:11:43,780 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:11:43,780 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:11:43,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:32953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:11:44,065 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:11:44.560215 28345 mini_ranger_kms.cc:310] Created kms service
I20260925 19:11:44.769486 28345 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:11:44.812124 28345 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:11:44 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:45.200495 28345 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:11:45.200605 28345 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwy_z0T/build/release/bin/postgresql.jar:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwy_z0T/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:11:45.200626 28345 mini_ranger_kms.cc:240] Using host: 127.27.174.84
I20260925 19:11:45.204512 28345 mini_ranger_kms.cc:276] Ranger KMS PID: 7994
I20260925 19:11:45.204609 28345 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.174.84:47681
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 19:11:47.450448 28345 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.212s user 0.002s sys 0.149s
I20260925 19:11:47.450558 28345 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:11:47.450606 28345 mini_ranger_kms.cc:401] 127.27.174.84:47681/kms/v1/keys
Sep 25 19:11:47 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 19:11:48 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
Sep 25 19:11:48 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
Sep 25 19:11:50 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:11:50.616447 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37149
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37149
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:50.700091 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:50.700295 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:50.700317 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:50.701572 8045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:50.701615 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:50.701629 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:50.701642 8045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:50.701655 8045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:50.703219 8045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.174.84:47681/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37149
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:50.703475 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:50.703751 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:50.709031 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:50.709077 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:50.709075 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:50.709717 8045 server_base.cc:1034] running on GCE node
I20260925 19:11:50.710001 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:50.710301 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:50.711529 8045 hybrid_clock.cc:648] HybridClock initialized: now 1790363510711467 us; error 47 us; skew 500 ppm
Sep 25 19:11:50 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:50.725240 8045 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:11:50.725910 8045 webserver.cc:460] Webserver started at http://127.27.174.126:45159/ using document root <none> and password file <none>
I20260925 19:11:50.726159 8045 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:50.726221 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:50.726369 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:50 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:50.784443 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "5a1238f6243649648c048602d8f44792"
format_stamp: "Formatted at 2026-09-25 19:11:50 on dist-test-slave-fcdn"
server_key: "903d663182715979093695a7855321aa"
server_key_iv: "12f8189ca51e65e811433ec8c4d54f03"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:50.784950 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "5a1238f6243649648c048602d8f44792"
format_stamp: "Formatted at 2026-09-25 19:11:50 on dist-test-slave-fcdn"
server_key: "903d663182715979093695a7855321aa"
server_key_iv: "12f8189ca51e65e811433ec8c4d54f03"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:50.786749 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 19:11:50.801433 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:50.801704 8045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:50.801815 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "5a1238f6243649648c048602d8f44792"
format_stamp: "Formatted at 2026-09-25 19:11:50 on dist-test-slave-fcdn"
server_key: "903d663182715979093695a7855321aa"
server_key_iv: "12f8189ca51e65e811433ec8c4d54f03"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:50.801913 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:50.825824 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:50.828330 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:50.828506 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:50.833575 8045 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:37149
I20260925 19:11:50.833676 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:37149 every 8 connection(s)
I20260925 19:11:50.834029 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:11:50.834779 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:50.837174 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Bootstrap starting.
I20260925 19:11:50.837777 8113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:50.838070 8113 log.cc:824] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:50.838776 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: No bootstrap required, opened a new log
I20260925 19:11:50.840816 8113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:50.840987 8113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:50.841033 8113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1238f6243649648c048602d8f44792, State: Initialized, Role: FOLLOWER
I20260925 19:11:50.841195 8113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:50.841290 8113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:11:50.841334 8113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:11:50.841382 8113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:50.841908 8113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:50.842028 8113 leader_election.cc:302] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a1238f6243649648c048602d8f44792; no voters:
I20260925 19:11:50.842232 8113 leader_election.cc:288] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:11:50.842259 8116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:50.842377 8116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1238f6243649648c048602d8f44792, State: Running, Role: LEADER
I20260925 19:11:50.842494 8113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:50.842518 8116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:50.842994 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8045
I20260925 19:11:50.843088 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:11:50.843246 28345 external_mini_cluster.cc:1428] Setting key ba174c1ba85b7353231cbf8daf790b80
I20260925 19:11:50.843842 8117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a1238f6243649648c048602d8f44792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } } }
I20260925 19:11:50.843919 8117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:50.844043 8113 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:11:50.844214 8118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a1238f6243649648c048602d8f44792. Latest consensus state: current_term: 1 leader_uuid: "5a1238f6243649648c048602d8f44792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } } }
I20260925 19:11:50.844322 8118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:50.844460 8132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:11:50.847223 8132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:11:50.849721 8132 catalog_manager.cc:1347] Generated new cluster ID: e627fc8a763d4074866b86e4dbd31f1e
I20260925 19:11:50.849823 8132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:11:50 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:50.859818 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:50.844062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.27.174.126:37149)
0925 19:11:50.845978 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:50.846026 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:50.846112 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:11:50.846526 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:50.846541 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:50.846686 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:50.846896 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:50.848258 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:50.849925 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:50.851182 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:50.851907 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:50.852018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:50.855211 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:50.855215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:50.855216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:50.855228 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:50.857482 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:50.857918 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:50.857922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:50.857922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:50.858046 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:50.858276 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:50.858277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:50.858278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:50.858621 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:50.858776 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:50.859362 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 19:11:50.859501 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":90,"threads_started":1}
I20260925 19:11:50.873696 8132 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:11:50.874552 8132 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:11:50.881217 8132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Generated new TSK 0
I20260925 19:11:50.881495 8132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:11:51.552591 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:51.667382 8158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:51.667690 8158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:51.667729 8158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:51.669636 8158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:51.669700 8158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:51.669742 8158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:11:51.671352 8158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:51.671607 8158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:51.671944 8158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:51.676966 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:51.677120 8163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:51.677345 8166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:51.677378 8158 server_base.cc:1034] running on GCE node
I20260925 19:11:51.677645 8158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:51.677935 8158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:51.679133 8158 hybrid_clock.cc:648] HybridClock initialized: now 1790363511679114 us; error 31 us; skew 500 ppm
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:51.682195 8158 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:11:51.682790 8158 webserver.cc:460] Webserver started at http://127.27.174.65:39251/ using document root <none> and password file <none>
I20260925 19:11:51.683040 8158 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:51.683109 8158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:51.683259 8158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:51.697079 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "549743a2467b4549a24f0827792524bd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "f724b651d68b4b052eeb1474fb3d9677"
server_key_iv: "8e09531a1fe06c6e8a759277e6229740"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.697597 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "549743a2467b4549a24f0827792524bd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "f724b651d68b4b052eeb1474fb3d9677"
server_key_iv: "8e09531a1fe06c6e8a759277e6229740"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.699698 8158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:11:51.720006 8175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:51.720264 8158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 19:11:51.720366 8158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "549743a2467b4549a24f0827792524bd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "f724b651d68b4b052eeb1474fb3d9677"
server_key_iv: "8e09531a1fe06c6e8a759277e6229740"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.720469 8158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:51.747103 8158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:51.749717 8158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:51.749928 8158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:51.750157 8158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:51.750535 8158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:51.750582 8158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:51.750613 8158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:51.750661 8158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:51.757597 8158 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37665
I20260925 19:11:51.758030 8158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:11:51.759929 8288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37665 every 8 connection(s)
I20260925 19:11:51.763816 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8158
I20260925 19:11:51.763900 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:11:51.764091 28345 external_mini_cluster.cc:1428] Setting key dd0e9c7bfca1612f04c13e5ed117bc5d
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:51.771862 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:51.758781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:56099 (local address 127.27.174.126:37149)
0925 19:11:51.761686 (+ 2905us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:51.761689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:51.761702 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:11:51.762070 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:51.762071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:51.762084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:51.762180 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:51.762898 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:51.763718 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:51.765402 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:51.765734 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:51.765799 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:51.768457 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:51.768458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:51.768459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:51.768466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:51.770285 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:51.770777 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:51.770779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:51.770779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:51.770839 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:51.771097 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:51.771097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:51.771098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:51.771238 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:51.771298 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:51.771669 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 19:11:51.771756 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2868,"thread_start_us":80,"threads_started":1}
I20260925 19:11:51.772861 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:51.758749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.65:56099)
0925 19:11:51.761225 (+ 2476us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:51.761274 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:51.761935 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:51.762278 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:51.762282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:51.762524 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:51.762784 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:51.762790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:51.763825 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:51.763826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:51.765307 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:51.765310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:51.765927 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:51.765929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:51.766077 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:51.766685 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:51.766695 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:51.768346 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:51.770379 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:51.770386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:51.770388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:51.770693 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:51.770899 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:51.770899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:51.770900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:51.771028 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:51.771324 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:51.771330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:51.771502 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 19:11:51.772388 (+ 886us) client_negotiation.cc:240] Negotiation successful
0925 19:11:51.772628 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2400,"thread_start_us":78,"threads_started":1}
I20260925 19:11:51.777657 8289 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:51.777768 8289 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:51.777959 8289 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:51.778574 8077 ts_manager.cc:194] Registered new tserver with Master: 549743a2467b4549a24f0827792524bd (127.27.174.65:37665)
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:11:51.779464 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:56099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:11:51.817119 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 19:11:51.938395 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:51.938681 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:51.938717 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:51.940469 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:51.940572 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:51.940630 8297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:11:51.942931 8297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:51.943220 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:51.943579 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:51.949997 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:51.950131 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:51.950073 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:51.950351 8297 server_base.cc:1034] running on GCE node
I20260925 19:11:51.950660 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:51.950937 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:51.952111 8297 hybrid_clock.cc:648] HybridClock initialized: now 1790363511952089 us; error 34 us; skew 500 ppm
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:51.955328 8297 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:11:51.955859 8297 webserver.cc:460] Webserver started at http://127.27.174.66:32905/ using document root <none> and password file <none>
I20260925 19:11:51.956120 8297 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:51.956195 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:51.956349 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:51 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:51.973645 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "cde072876f694a20a2506bfa91e584dd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "e9de9a79e05ad180b83a6a796df712c3"
server_key_iv: "cca3f7d9eeef2c86eaa7310b004c7b04"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.974131 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "cde072876f694a20a2506bfa91e584dd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "e9de9a79e05ad180b83a6a796df712c3"
server_key_iv: "cca3f7d9eeef2c86eaa7310b004c7b04"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.976004 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 19:11:51.989801 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:51.990020 8297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:51.990135 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "cde072876f694a20a2506bfa91e584dd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "e9de9a79e05ad180b83a6a796df712c3"
server_key_iv: "cca3f7d9eeef2c86eaa7310b004c7b04"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:51.990226 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:52.004406 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:52.007062 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:52.007273 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:52.007568 8297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:52.008005 8297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:52.008107 8297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.008179 8297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:52.008237 8297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.014482 8297 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44169
I20260925 19:11:52.014956 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:11:52.017391 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8297
I20260925 19:11:52.017498 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:11:52.017704 28345 external_mini_cluster.cc:1428] Setting key c3f4b053ca70fbaa9210405347dd38e9
I20260925 19:11:52.018299 8426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44169 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:52.044972 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.019237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:50533 (local address 127.27.174.126:37149)
0925 19:11:52.019291 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:52.019293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:52.022920 (+ 3627us) server_negotiation.cc:407] Connection header received
0925 19:11:52.024523 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:52.024524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:52.024553 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:52.024587 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:52.035783 (+ 11196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.036610 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.038006 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.038353 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.038422 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.041398 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:52.041400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:52.041400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:52.041409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:52.043262 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.043828 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.043830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.043830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.043889 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.044191 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.044192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.044192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.044331 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:52.044390 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:52.044771 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 19:11:52.044859 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:11:52.045733 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.018920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.66:50533)
0925 19:11:52.022770 (+ 3850us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:52.022823 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:52.023501 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:52.024785 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:52.024790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:52.025044 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:52.025321 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.025327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.036789 (+ 11462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.036790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.037897 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.037900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.038510 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.038512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.038669 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.039338 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:52.039349 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:52.041272 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:52.043369 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.043377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.043380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.043730 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.043962 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.043963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.043963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.044116 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.044429 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:52.044435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:52.044609 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 19:11:52.045327 (+ 718us) client_negotiation.cc:240] Negotiation successful
0925 19:11:52.045496 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3798,"thread_start_us":83,"threads_started":1}
I20260925 19:11:52.046356 8427 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:52.046451 8427 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.046643 8427 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:52.047080 8077 ts_manager.cc:194] Registered new tserver with Master: cde072876f694a20a2506bfa91e584dd (127.27.174.66:44169)
I20260925 19:11:52.047561 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:50533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:11:52.055541 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:52.169426 8432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:52.169728 8432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:52.169764 8432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:52.171423 8432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:52.171485 8432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:52.171525 8432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:11:52.173688 8432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:52.173949 8432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:52.174237 8432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:52.179747 8437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.179847 8432 server_base.cc:1034] running on GCE node
W20260925 19:11:52.179843 8438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:52.179767 8440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.180349 8432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:52.180668 8432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:52.181871 8432 hybrid_clock.cc:648] HybridClock initialized: now 1790363512181841 us; error 42 us; skew 500 ppm
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:52.185257 8432 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:11:52.185709 8432 webserver.cc:460] Webserver started at http://127.27.174.67:41897/ using document root <none> and password file <none>
I20260925 19:11:52.186012 8432 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:52.186102 8432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:52.186245 8432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:52.201220 8432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "880d68345d524054a089f53f92ded76a"
format_stamp: "Formatted at 2026-09-25 19:11:52 on dist-test-slave-fcdn"
server_key: "2957ee0c9db1a2357bd863d450888352"
server_key_iv: "48ccdd1129a38336557ac2a70884003d"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.201800 8432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "880d68345d524054a089f53f92ded76a"
format_stamp: "Formatted at 2026-09-25 19:11:52 on dist-test-slave-fcdn"
server_key: "2957ee0c9db1a2357bd863d450888352"
server_key_iv: "48ccdd1129a38336557ac2a70884003d"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.204350 8432 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 19:11:52.221725 8447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.221997 8432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 19:11:52.222110 8432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "880d68345d524054a089f53f92ded76a"
format_stamp: "Formatted at 2026-09-25 19:11:52 on dist-test-slave-fcdn"
server_key: "2957ee0c9db1a2357bd863d450888352"
server_key_iv: "48ccdd1129a38336557ac2a70884003d"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.222203 8432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:52.245230 8432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:52.247764 8432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:52.247990 8432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:52.248262 8432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:52.248667 8432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:52.248728 8432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.248796 8432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:52.248826 8432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.256072 8432 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:39603
I20260925 19:11:52.256501 8432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:11:52.259290 8560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:39603 every 8 connection(s)
I20260925 19:11:52.266181 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8432
I20260925 19:11:52.266289 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:11:52.266481 28345 external_mini_cluster.cc:1428] Setting key 037dc426b79b881f51f249fe7aa2a978
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:52.272584 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.260298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:42467 (local address 127.27.174.126:37149)
0925 19:11:52.260351 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:52.260353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:52.260673 (+ 320us) server_negotiation.cc:407] Connection header received
0925 19:11:52.261331 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:52.261332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:52.261345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:52.261427 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:52.262217 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.263001 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.264593 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.264898 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.264983 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.269007 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:52.269009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:52.269009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:52.269018 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:52.270782 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.271320 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.271322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.271322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.271389 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.271757 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.271758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.271758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.271910 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:52.271988 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:52.272377 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 19:11:52.272477 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:11:52.272743 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.259957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.67:42467)
0925 19:11:52.260506 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:52.260553 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:52.261192 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:52.261537 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:52.261542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:52.261793 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:52.262092 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.262099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.263141 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.263142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.264470 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.264473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.265039 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.265041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.265195 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.267070 (+ 1875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:52.267081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:52.268892 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:52.270888 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.270896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.270899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.271221 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.271483 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.271484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.271484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.271678 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.272003 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:52.272010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:52.272186 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 19:11:52.272302 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 19:11:52.272497 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":92,"threads_started":1}
I20260925 19:11:52.273432 8561 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:52.273550 8561 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.273764 8561 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:52.274163 8077 ts_manager.cc:194] Registered new tserver with Master: 880d68345d524054a089f53f92ded76a (127.27.174.67:39603)
I20260925 19:11:52.274713 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:42467
I20260925 19:11:52.278337 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:11:52.279119 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8158
I20260925 19:11:52.286341 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8297
I20260925 19:11:52.292734 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8432
I20260925 19:11:52.299078 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8045
I20260925 19:11:52.304132 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37149
--webserver_interface=127.27.174.126
--webserver_port=45159
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37149
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:52.386632 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:52.386832 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:52.386853 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:52.388069 8564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:52.388114 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:52.388129 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:52.388141 8564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:52.388154 8564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:52.389747 8564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37149
--ranger_config_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:52.390004 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:52.390235 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:52.395503 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:52.395444 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.395748 8564 server_base.cc:1034] running on GCE node
W20260925 19:11:52.395457 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.396080 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:52.396334 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:52.397490 8564 hybrid_clock.cc:648] HybridClock initialized: now 1790363512397480 us; error 29 us; skew 500 ppm
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:52.400197 8564 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:11:52.400657 8564 webserver.cc:460] Webserver started at http://127.27.174.126:45159/ using document root <none> and password file <none>
I20260925 19:11:52.400840 8564 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:52.400895 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:52.462194 8564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:52.462930 8580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.463150 8564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:52.463258 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "5a1238f6243649648c048602d8f44792"
format_stamp: "Formatted at 2026-09-25 19:11:50 on dist-test-slave-fcdn"
server_key: "903d663182715979093695a7855321aa"
server_key_iv: "12f8189ca51e65e811433ec8c4d54f03"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.463713 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:52.483827 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:52.485960 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:52.486135 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:52.490541 8564 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:37149
I20260925 19:11:52.490585 8632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:37149 every 8 connection(s)
I20260925 19:11:52.490974 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:11:52.492241 8633 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:11:52.492925 8633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Bootstrap starting.
I20260925 19:11:52.500037 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8564
I20260925 19:11:52.500379 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:37665
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:37149
--webserver_port=39251
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:11:52.501108 8633 log.cc:824] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:52.502929 8633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:11:52.503226 8633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Bootstrap complete.
I20260925 19:11:52.505693 8633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:52.505921 8633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1238f6243649648c048602d8f44792, State: Initialized, Role: FOLLOWER
I20260925 19:11:52.506098 8633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:52.506172 8633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:11:52.506237 8633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:11:52.506276 8633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:11:52.507261 8633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:52.507346 8633 leader_election.cc:302] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a1238f6243649648c048602d8f44792; no voters:
I20260925 19:11:52.507519 8633 leader_election.cc:288] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:11:52.507575 8637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:11:52.507736 8637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [term 2 LEADER]: Becoming Leader. State: Replica: 5a1238f6243649648c048602d8f44792, State: Running, Role: LEADER
I20260925 19:11:52.507835 8633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:52.507879 8637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } }
I20260925 19:11:52.508206 8638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5a1238f6243649648c048602d8f44792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } } }
I20260925 19:11:52.508271 8638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:52.508234 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a1238f6243649648c048602d8f44792. Latest consensus state: current_term: 2 leader_uuid: "5a1238f6243649648c048602d8f44792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a1238f6243649648c048602d8f44792" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37149 } } }
I20260925 19:11:52.508327 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:52.508445 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:11:52.508991 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:11:52.509198 8641 catalog_manager.cc:1259] Loaded cluster ID: e627fc8a763d4074866b86e4dbd31f1e
I20260925 19:11:52.509292 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:11:52.509619 8633 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:11:52.513147 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:11:52.513815 8641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Loaded TSK: 0
I20260925 19:11:52.515518 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:11:52.614336 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:52.614629 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:52.614681 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:52.616432 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:52.616495 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:52.616537 8636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:11:52.618685 8636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:37665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=39251
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:52.618952 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:52.619223 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:52.625834 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.625944 8636 server_base.cc:1034] running on GCE node
W20260925 19:11:52.626186 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:52.626353 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.626560 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:52.626830 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:52.628026 8636 hybrid_clock.cc:648] HybridClock initialized: now 1790363512628004 us; error 35 us; skew 500 ppm
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:52.631106 8636 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:11:52.631688 8636 webserver.cc:460] Webserver started at http://127.27.174.65:39251/ using document root <none> and password file <none>
I20260925 19:11:52.632017 8636 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:52.632153 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:52.654333 8636 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:11:52.655431 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.655699 8636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:11:52.655782 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "549743a2467b4549a24f0827792524bd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "f724b651d68b4b052eeb1474fb3d9677"
server_key_iv: "8e09531a1fe06c6e8a759277e6229740"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.656248 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:52.676231 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:52.678905 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:52.679121 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:52.679392 8636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:52.679869 8636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:52.679976 8636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.680079 8636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:52.680154 8636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.686086 8636 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37665
I20260925 19:11:52.686553 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:11:52.689244 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37665 every 8 connection(s)
I20260925 19:11:52.691792 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8636
I20260925 19:11:52.692090 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:44169
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:37149
--webserver_port=32905
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:52.703020 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.689900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.65:34561)
0925 19:11:52.690282 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:52.690323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:52.690916 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:52.692237 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:52.692241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:52.692476 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:52.692748 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.692754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.694535 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.694536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.695928 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.695931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.696731 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.696732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.696862 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.697468 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:52.697477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:52.699098 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:52.701245 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.701251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.701253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.701515 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.701766 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.701767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.701767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.701900 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.702395 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:52.702401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:52.702546 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 19:11:52.702639 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:11:52.702793 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":93,"threads_started":1}
I20260925 19:11:52.704027 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.689951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34561 (local address 127.27.174.126:37149)
0925 19:11:52.691773 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:52.691782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:52.691823 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:11:52.691971 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:52.691978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:52.692032 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:52.692891 (+ 859us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:52.692972 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.694406 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.696040 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.696606 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.697151 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.699215 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:52.699219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:52.699219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:52.699231 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:52.701150 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.701601 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.701604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.701604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.701695 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.701977 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.701977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.701978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.702307 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:52.703650 (+ 1343us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:52.703723 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:11:52.703835 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1759,"thread_start_us":83,"threads_started":1}
I20260925 19:11:52.704972 8795 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:52.705057 8795 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.705228 8795 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:52.706096 8795 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:37149 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:11:52.706162 8795 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.706228 8795 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:52.706640 8795 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.706714 8795 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:52.707043 8795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:11:52.810623 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:52.810861 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:52.810901 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:52.812258 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:52.812326 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:52.812387 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:11:52.814198 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:44169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=32905
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:52.814452 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:52.814699 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:52.821053 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:52.821069 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:52.821385 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:52.822351 8799 server_base.cc:1034] running on GCE node
I20260925 19:11:52.822597 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:52.822866 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:52.830278 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790363512830246 us; error 39 us; skew 500 ppm
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:52.833312 8799 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:11:52.833755 8799 webserver.cc:460] Webserver started at http://127.27.174.66:32905/ using document root <none> and password file <none>
I20260925 19:11:52.833997 8799 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:52.834081 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:52.851357 8799 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 19:11:52.852406 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.852631 8799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:11:52.852731 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "cde072876f694a20a2506bfa91e584dd"
format_stamp: "Formatted at 2026-09-25 19:11:51 on dist-test-slave-fcdn"
server_key: "e9de9a79e05ad180b83a6a796df712c3"
server_key_iv: "cca3f7d9eeef2c86eaa7310b004c7b04"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:52.853188 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:52.881259 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:52.883461 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:52.883579 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:52.883900 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:52.884302 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:52.884341 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.884364 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:52.884377 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:52.891351 8799 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44169
I20260925 19:11:52.891917 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:11:52.893347 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44169 every 8 connection(s)
I20260925 19:11:52.894507 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8799
I20260925 19:11:52.894804 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:47681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:39603
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:37149
--webserver_port=41897
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:11:52 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363512, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:52.904733 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.894164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51697 (local address 127.27.174.126:37149)
0925 19:11:52.894225 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:52.894228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:52.894424 (+ 196us) server_negotiation.cc:407] Connection header received
0925 19:11:52.895228 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:52.895229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:52.895246 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:52.895278 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:52.896154 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.896992 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.898512 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.898823 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.898866 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.901475 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:52.901477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:52.901477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:52.901485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:52.903184 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.903692 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.903694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.903694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.903752 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:52.904021 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:52.904022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:52.904022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:52.904151 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:52.904200 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:52.904548 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 19:11:52.904633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:11:52.905566 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:52.894138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.66:51697)
0925 19:11:52.894287 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:52.894331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:52.895097 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:52.895476 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:52.895481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:52.895760 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:52.896040 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.896046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.897115 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.897117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.898393 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:52.898395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:52.898973 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:52.898974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:52.899107 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:52.899718 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:52.899729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:52.901354 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:52.903276 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.903283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.903285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.903581 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.903823 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:52.903824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:52.903824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:52.903949 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:52.904231 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:52.904237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:52.904390 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 19:11:52.905161 (+ 771us) client_negotiation.cc:240] Negotiation successful
0925 19:11:52.905323 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 19:11:52.906222 8929 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:52.906324 8929 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.906517 8929 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:52.907193 8929 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:37149 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:11:52.907282 8929 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.907353 8929 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:52.907684 8929 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:52.907759 8929 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:52.908042 8929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:11:53.025317 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:53.025606 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:53.025655 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:53.027400 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:53.027454 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:53.027493 8932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:11:53.029593 8932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:47681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:39603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=41897
--tserver_master_addrs=127.27.174.126:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:53.029822 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:53.030053 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:53.035259 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:53.035260 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:53.035254 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:53.035472 8932 server_base.cc:1034] running on GCE node
I20260925 19:11:53.035708 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:53.035959 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:53.037132 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790363513037125 us; error 40 us; skew 500 ppm
Sep 25 19:11:53 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:53.039848 8932 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:11:53.040278 8932 webserver.cc:460] Webserver started at http://127.27.174.67:41897/ using document root <none> and password file <none>
I20260925 19:11:53.040529 8932 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:53.040613 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:11:53 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:53.059700 8932 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 19:11:53.065076 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:53.067081 8932 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260925 19:11:53.067270 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "880d68345d524054a089f53f92ded76a"
format_stamp: "Formatted at 2026-09-25 19:11:52 on dist-test-slave-fcdn"
server_key: "2957ee0c9db1a2357bd863d450888352"
server_key_iv: "48ccdd1129a38336557ac2a70884003d"
server_key_version: "kuduclusterkey@0"
I20260925 19:11:53.067910 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:53.086467 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:53.088723 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:53.088891 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:53.089170 8932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:11:53.089557 8932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:11:53.089605 8932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:53.089637 8932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:11:53.089661 8932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:53.097146 8932 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:39603
I20260925 19:11:53.097666 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:11:53.098085 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:39603 every 8 connection(s)
I20260925 19:11:53.101931 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 8932
Sep 25 19:11:53 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:53.115335 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:53.100311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:54167 (local address 127.27.174.126:37149)
0925 19:11:53.100387 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:53.100390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:53.100629 (+ 239us) server_negotiation.cc:407] Connection header received
0925 19:11:53.101216 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:53.101218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:53.101230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:53.101313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:53.102151 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.102926 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.104464 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.104669 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.104746 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:53.111691 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:53.111692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:53.111693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:53.111700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:53.113641 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:53.114186 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:53.114187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:53.114187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:53.114252 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:53.114613 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:53.114614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:53.114614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:53.114727 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:53.114838 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:53.115198 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 19:11:53.115267 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:11:53.115671 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:53.100297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37149 (local address 127.27.174.67:54167)
0925 19:11:53.100471 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:53.100514 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:53.101095 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:53.101404 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:53.101409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:53.101661 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:53.101963 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:53.101970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.103066 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.103067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:53.104378 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:53.104380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.104783 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.104784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:53.104910 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:53.105489 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:53.105500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:53.111546 (+ 6046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:53.113744 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:53.113755 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:53.113759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:53.114115 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:53.114347 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:53.114349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:53.114349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:53.114530 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:53.114847 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:53.114855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:53.115046 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 19:11:53.115178 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 19:11:53.115359 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260925 19:11:53.116266 9063 heartbeater.cc:347] Connected to a master server at 127.27.174.126:37149
I20260925 19:11:53.116382 9063 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:53.116557 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:53.102339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.27.174.126:37149)
0925 19:11:53.103086 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:53.103088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:53.103295 (+ 207us) server_negotiation.cc:407] Connection header received
0925 19:11:53.103318 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:53.103319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:53.103333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:53.103400 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:53.104209 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.104950 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.106390 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:53.106596 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:53.106683 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:53.111441 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:53.111443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:53.111443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:53.111452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:53.113453 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:53.114636 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:53.114638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:53.114638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:53.114700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:53.115799 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:53.115800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:53.115800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:53.115933 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:53.116106 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:53.116360 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 19:11:53.116461 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":63,"threads_started":1}
I20260925 19:11:53.116608 9063 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:53.117394 9063 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:37149 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:11:53.117482 9063 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:53.117561 9063 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:53.117890 9063 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:53.117965 9063 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
W20260925 19:11:53.118284 9063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 19:11:53 dist-test-slave-fcdn krb5kdc[7601](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:53.707242 8795 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:53.707419 8795 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:53.707808 8597 ts_manager.cc:194] Registered new tserver with Master: 549743a2467b4549a24f0827792524bd (127.27.174.65:37665)
I20260925 19:11:53.708896 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:34561
I20260925 19:11:53.908216 8929 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:53.908373 8929 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:53.908831 8597 ts_manager.cc:194] Registered new tserver with Master: cde072876f694a20a2506bfa91e584dd (127.27.174.66:44169)
I20260925 19:11:53.909382 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:51697
Sep 25 19:11:53 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:54.118485 9063 heartbeater.cc:464] Registering TS with master...
I20260925 19:11:54.118645 9063 heartbeater.cc:511] Master 127.27.174.126:37149 requested a full tablet report, sending...
I20260925 19:11:54.119007 8597 ts_manager.cc:194] Registered new tserver with Master: 880d68345d524054a089f53f92ded76a (127.27.174.67:39603)
I20260925 19:11:54.119526 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:54167
I20260925 19:11:54.124797 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:11:54.140031 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.125797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.27.174.126:37149)
0925 19:11:54.131967 (+ 6170us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.131969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.131977 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:11:54.132003 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.132003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.132016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.132084 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:54.132933 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.133718 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.134787 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.135114 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.135186 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.136795 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:54.136797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:54.136797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:54.136806 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:54.138688 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:54.139122 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:54.139124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:54.139124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:54.139197 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:54.139408 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:54.139409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:54.139409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:54.139559 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:54.139683 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.139829 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.139922 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6147,"thread_start_us":66,"threads_started":1}
I20260925 19:11:54.142539 8597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:11:54.145069 8597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:11:54.153401 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:39603 (local address 127.0.0.1:40954)
0925 19:11:54.148810 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.148855 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:54.148971 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:54.149272 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:54.149274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:54.149277 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:54.149357 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.149359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.150561 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.150562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.151914 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.151917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.153085 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.153086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.153180 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.153195 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 19:11:54.153265 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:11:54.153317 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":82,"threads_started":1}
I20260925 19:11:54.153947 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.27.174.67:39603)
0925 19:11:54.148492 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.148497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.148927 (+ 430us) server_negotiation.cc:407] Connection header received
0925 19:11:54.149080 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.149083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.149099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.149154 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:54.149441 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.150441 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.152022 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.152910 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.152961 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.153019 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.153823 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.153868 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260925 19:11:54.154649 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37665 (local address 127.0.0.1:46052)
0925 19:11:54.149567 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.149575 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:54.149609 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:54.150014 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:54.150014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:54.150016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:54.150061 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.150062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.151651 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.151651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.152845 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.152847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.154372 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.154373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.154451 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.154456 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:11:54.154529 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:11:54.154573 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":64,"threads_started":1}
I20260925 19:11:54.154667 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.27.174.65:37665)
0925 19:11:54.149376 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.149382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.149658 (+ 276us) server_negotiation.cc:407] Connection header received
0925 19:11:54.149693 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.149695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.149846 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.149931 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:54.150135 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.151536 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.152969 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.154076 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.154150 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.154336 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.154549 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.154593 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":53,"threads_started":1}
I20260925 19:11:54.154899 8979 tablet_service.cc:1467] Processing CreateTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 (DEFAULT_TABLE table=test-table [id=5b0adc1dba284d2da66af00a7438faf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:54.155385 8979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f02b0bf8cd4895a314e81eca61e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:54.155845 8729 tablet_service.cc:1467] Processing CreateTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 (DEFAULT_TABLE table=test-table [id=5b0adc1dba284d2da66af00a7438faf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:54.156334 8729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f02b0bf8cd4895a314e81eca61e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:54.159049 9096 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Bootstrap starting.
I20260925 19:11:54.159919 9096 tablet_bootstrap.cc:654] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:54.160676 9096 log.cc:824] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:54.161134 9097 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Bootstrap starting.
I20260925 19:11:54.161717 9096 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: No bootstrap required, opened a new log
I20260925 19:11:54.161816 9096 ts_tablet_manager.cc:1397] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:11:54.164234 9096 raft_consensus.cc:348] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.164449 9096 raft_consensus.cc:374] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:54.164489 9096 raft_consensus.cc:729] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880d68345d524054a089f53f92ded76a, State: Initialized, Role: FOLLOWER
I20260925 19:11:54.165141 9096 consensus_queue.cc:260] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.165436 9096 ts_tablet_manager.cc:1428] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 19:11:54.165558 9063 heartbeater.cc:503] Master 127.27.174.126:37149 was elected leader, sending a full tablet report...
I20260925 19:11:54.166978 9097 tablet_bootstrap.cc:654] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:54.167395 9097 log.cc:824] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:54.168871 9097 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: No bootstrap required, opened a new log
I20260925 19:11:54.169005 9097 ts_tablet_manager.cc:1397] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260925 19:11:54.170473 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44169 (local address 127.0.0.1:33376)
0925 19:11:54.149435 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.149446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:54.149495 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:54.166589 (+ 17094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:54.166590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:54.166593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:54.166656 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.166658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.168088 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.168089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.169226 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.169229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.170224 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.170225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.170291 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.170297 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:11:54.170331 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 19:11:54.170382 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":88,"threads_started":1}
I20260925 19:11:54.170828 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.148584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.27.174.66:44169)
0925 19:11:54.166399 (+ 17815us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.166405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.166418 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:11:54.166454 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.166456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.166474 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.166591 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:54.166744 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.167978 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.169339 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.170111 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.170606 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.170678 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.170708 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.170751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17770,"thread_start_us":59,"threads_started":1}
I20260925 19:11:54.171762 8859 tablet_service.cc:1467] Processing CreateTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 (DEFAULT_TABLE table=test-table [id=5b0adc1dba284d2da66af00a7438faf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:11:54.172112 8859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f02b0bf8cd4895a314e81eca61e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:54.172693 9097 raft_consensus.cc:348] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.172863 9097 raft_consensus.cc:374] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:54.172914 9097 raft_consensus.cc:729] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549743a2467b4549a24f0827792524bd, State: Initialized, Role: FOLLOWER
I20260925 19:11:54.173050 9097 consensus_queue.cc:260] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.173352 9097 ts_tablet_manager.cc:1428] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260925 19:11:54.173496 8795 heartbeater.cc:503] Master 127.27.174.126:37149 was elected leader, sending a full tablet report...
I20260925 19:11:54.181869 9102 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Bootstrap starting.
I20260925 19:11:54.182739 9102 tablet_bootstrap.cc:654] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:54.183168 9102 log.cc:824] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:54.184115 9102 tablet_bootstrap.cc:492] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: No bootstrap required, opened a new log
I20260925 19:11:54.184193 9102 ts_tablet_manager.cc:1397] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:11:54.186833 9102 raft_consensus.cc:348] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.187023 9102 raft_consensus.cc:374] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:54.187059 9102 raft_consensus.cc:729] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde072876f694a20a2506bfa91e584dd, State: Initialized, Role: FOLLOWER
I20260925 19:11:54.187328 9102 consensus_queue.cc:260] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.187700 9102 ts_tablet_manager.cc:1428] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:11:54.187870 8929 heartbeater.cc:503] Master 127.27.174.126:37149 was elected leader, sending a full tablet report...
W20260925 19:11:54.191183 8796 tablet.cc:2369] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:11:54 dist-test-slave-fcdn krb5kdc[7601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20260925 19:11:54.252651 9099 raft_consensus.cc:482] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:11:54.252803 9099 raft_consensus.cc:504] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.253197 9099 leader_election.cc:288] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde072876f694a20a2506bfa91e584dd (127.27.174.66:44169), 549743a2467b4549a24f0827792524bd (127.27.174.65:37665)
I20260925 19:11:54.260020 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.253717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37227 (local address 127.27.174.65:37665)
0925 19:11:54.253758 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.253759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.255013 (+ 1254us) server_negotiation.cc:407] Connection header received
0925 19:11:54.255108 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.255109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.255119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.255157 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:54.255352 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.256054 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.258406 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.259059 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.259120 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.259131 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.259857 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.259914 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:11:54.260922 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.253425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37665 (local address 127.27.174.67:37227)
0925 19:11:54.254939 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.254950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:54.255039 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:54.255218 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:54.255219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:54.255220 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:54.255283 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.255284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.256642 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.256644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.258300 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.258303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.259660 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.259661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.259732 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.259736 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:11:54.260376 (+ 640us) client_negotiation.cc:240] Negotiation successful
0925 19:11:54.260422 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":81,"threads_started":1}
I20260925 19:11:54.261502 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f02b0bf8cd4895a314e81eca61e5b0" candidate_uuid: "880d68345d524054a089f53f92ded76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549743a2467b4549a24f0827792524bd" is_pre_election: true
I20260925 19:11:54.261677 8749 raft_consensus.cc:2454] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880d68345d524054a089f53f92ded76a in term 0.
I20260925 19:11:54.262039 8951 leader_election.cc:302] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549743a2467b4549a24f0827792524bd, 880d68345d524054a089f53f92ded76a; no voters:
I20260925 19:11:54.262180 9099 raft_consensus.cc:2781] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:11:54.262216 9099 raft_consensus.cc:482] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:11:54.262238 9099 raft_consensus.cc:3037] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:54.262524 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.253602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:49395 (local address 127.27.174.66:44169)
0925 19:11:54.253646 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.253649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.253763 (+ 114us) server_negotiation.cc:407] Connection header received
0925 19:11:54.253866 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.253867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.253877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.253906 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:11:54.254263 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.255009 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.261317 (+ 6308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.261867 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.261923 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.261932 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.262392 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.262444 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:11:54.262765 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.253365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44169 (local address 127.27.174.67:49395)
0925 19:11:54.253703 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.253714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:54.253789 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:54.254102 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:54.254103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:54.254105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:11:54.254181 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.254183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.256075 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.256076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.260775 (+ 4699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:54.260778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.262229 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.262230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:54.262305 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.262309 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:11:54.262632 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 19:11:54.262685 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":114,"threads_started":1}
I20260925 19:11:54.263095 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f02b0bf8cd4895a314e81eca61e5b0" candidate_uuid: "880d68345d524054a089f53f92ded76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde072876f694a20a2506bfa91e584dd" is_pre_election: true
I20260925 19:11:54.263116 9099 raft_consensus.cc:504] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.263239 8868 raft_consensus.cc:2454] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880d68345d524054a089f53f92ded76a in term 0.
I20260925 19:11:54.263265 9099 leader_election.cc:288] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [CANDIDATE]: Term 1 election: Requested vote from peers cde072876f694a20a2506bfa91e584dd (127.27.174.66:44169), 549743a2467b4549a24f0827792524bd (127.27.174.65:37665)
I20260925 19:11:54.263460 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f02b0bf8cd4895a314e81eca61e5b0" candidate_uuid: "880d68345d524054a089f53f92ded76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde072876f694a20a2506bfa91e584dd"
I20260925 19:11:54.263532 8868 raft_consensus.cc:3037] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:54.264254 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f02b0bf8cd4895a314e81eca61e5b0" candidate_uuid: "880d68345d524054a089f53f92ded76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549743a2467b4549a24f0827792524bd"
I20260925 19:11:54.264343 8749 raft_consensus.cc:3037] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:54.264436 8868 raft_consensus.cc:2454] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880d68345d524054a089f53f92ded76a in term 1.
I20260925 19:11:54.264624 8950 leader_election.cc:302] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880d68345d524054a089f53f92ded76a, cde072876f694a20a2506bfa91e584dd; no voters:
I20260925 19:11:54.264775 9099 raft_consensus.cc:2781] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:54.264919 9099 raft_consensus.cc:686] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 1 LEADER]: Becoming Leader. State: Replica: 880d68345d524054a089f53f92ded76a, State: Running, Role: LEADER
I20260925 19:11:54.265018 9099 consensus_queue.cc:237] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } }
I20260925 19:11:54.265232 8749 raft_consensus.cc:2454] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880d68345d524054a089f53f92ded76a in term 1.
I20260925 19:11:54.265834 8597 catalog_manager.cc:5495] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a reported cstate change: term changed from 0 to 1, leader changed from <none> to 880d68345d524054a089f53f92ded76a (127.27.174.67). New cstate: current_term: 1 leader_uuid: "880d68345d524054a089f53f92ded76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880d68345d524054a089f53f92ded76a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 39603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 } health_report { overall_health: UNKNOWN } } }
I20260925 19:11:54.299285 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.296040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.27.174.67:39603)
0925 19:11:54.296089 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:54.296091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:54.296107 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:11:54.296153 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:54.296153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:54.296175 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:54.296266 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:11:54.296431 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.297135 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.298158 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:54.298435 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:54.298508 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:54.298716 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:11:54.298851 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:11:54.298923 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:54.299186 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 19:11:54.299207 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:11:54.304777 8868 raft_consensus.cc:1261] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 1 FOLLOWER]: Refusing update from remote peer 880d68345d524054a089f53f92ded76a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:54.304767 8749 raft_consensus.cc:1261] T 95f02b0bf8cd4895a314e81eca61e5b0 P 549743a2467b4549a24f0827792524bd [term 1 FOLLOWER]: Refusing update from remote peer 880d68345d524054a089f53f92ded76a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:54.305148 9109 consensus_queue.cc:1035] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "549743a2467b4549a24f0827792524bd" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:54.305354 9099 consensus_queue.cc:1035] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde072876f694a20a2506bfa91e584dd" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:54.310984 9112 mvcc.cc:205] Tried to move back new op lower bound from 7333328954589171712 to 7333328954430107648. Current Snapshot: MvccSnapshot[applied={T|T < 7333328954589171712}]
I20260925 19:11:54.312073 9115 mvcc.cc:205] Tried to move back new op lower bound from 7333328954589171712 to 7333328954430107648. Current Snapshot: MvccSnapshot[applied={T|T < 7333328954589171712}]
I20260925 19:11:54.314178 9113 mvcc.cc:205] Tried to move back new op lower bound from 7333328954589171712 to 7333328954430107648. Current Snapshot: MvccSnapshot[applied={T|T < 7333328954589171712}]
I20260925 19:11:54.317975 8597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:11:54.318084 8597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:11:54.318750 8597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a1238f6243649648c048602d8f44792: Sending DeleteTablet for 3 replicas of tablet 95f02b0bf8cd4895a314e81eca61e5b0
I20260925 19:11:54.319175 8979 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:54 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:40954
I20260925 19:11:54.319407 8859 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:54 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33376
I20260925 19:11:54.319790 8729 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f02b0bf8cd4895a314e81eca61e5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:11:54 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46052
I20260925 19:11:54.321012 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8636
I20260925 19:11:54.335763 9122 tablet_replica.cc:330] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: stopping tablet replica
I20260925 19:11:54.335732 9121 tablet_replica.cc:330] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: stopping tablet replica
I20260925 19:11:54.335880 9121 raft_consensus.cc:2229] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:11:54.336024 9121 raft_consensus.cc:2258] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:11:54.336567 9121 ts_tablet_manager.cc:1905] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:11:54.338541 9121 ts_tablet_manager.cc:1918] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:11:54.338621 9121 log.cc:1196] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/95f02b0bf8cd4895a314e81eca61e5b0
I20260925 19:11:54.338884 9121 ts_tablet_manager.cc:1939] T 95f02b0bf8cd4895a314e81eca61e5b0 P 880d68345d524054a089f53f92ded76a: Deleting consensus metadata
I20260925 19:11:54.339349 8581 catalog_manager.cc:4841] TS 880d68345d524054a089f53f92ded76a (127.27.174.67:39603): tablet 95f02b0bf8cd4895a314e81eca61e5b0 (table test-table [id=5b0adc1dba284d2da66af00a7438faf1]) successfully deleted
I20260925 19:11:54.342324 9122 raft_consensus.cc:2229] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:11:54.342543 9122 raft_consensus.cc:2258] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:11:54.346935 9122 ts_tablet_manager.cc:1905] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:11:54.349123 9122 ts_tablet_manager.cc:1918] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:11:54.349249 9122 log.cc:1196] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/95f02b0bf8cd4895a314e81eca61e5b0
I20260925 19:11:54.349568 9122 ts_tablet_manager.cc:1939] T 95f02b0bf8cd4895a314e81eca61e5b0 P cde072876f694a20a2506bfa91e584dd: Deleting consensus metadata
I20260925 19:11:54.351850 8581 catalog_manager.cc:4841] TS cde072876f694a20a2506bfa91e584dd (127.27.174.66:44169): tablet 95f02b0bf8cd4895a314e81eca61e5b0 (table test-table [id=5b0adc1dba284d2da66af00a7438faf1]) successfully deleted
W20260925 19:11:54.352226 8582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:37665 (error 108)
I20260925 19:11:54.353570 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8799
I20260925 19:11:54.354590 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:54.353703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37665 (local address 127.0.0.1:46064)
0925 19:11:54.354502 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:54.354555 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":776}
W20260925 19:11:54.355221 8582 catalog_manager.cc:4584] TS 549743a2467b4549a24f0827792524bd (127.27.174.65:37665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95f02b0bf8cd4895a314e81eca61e5b0: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37665: connect: Connection refused (error 111)
I20260925 19:11:54.360630 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8932
I20260925 19:11:54.367507 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 8564
I20260925 19:11:54.373232 28345 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:11:54.717748 28345 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:11:54.717832 28345 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:11:55.062754 28345 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:11:55.062 UTC [7631] LOG: received smart shutdown request
2026-09-25 19:11:55.066 UTC [7631] LOG: background worker "logical replication launcher" (PID 7640) exited with exit code 1
2026-09-25 19:11:55.067 UTC [7635] LOG: shutting down
2026-09-25 19:11:55.280 UTC [7631] LOG: database system is shut down
2026-09-25T19:11:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48189 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:11:55 dist-test-slave-fcdn krb5kdc[9133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:11:55 dist-test-slave-fcdn krb5kdc[9133](info): set up 2 sockets
Sep 25 19:11:55 dist-test-slave-fcdn krb5kdc[9133](info): commencing operation
krb5kdc: starting...
W20260925 19:11:56.452111 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:11:56.579087 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34637
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:56.661079 9146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:56.661350 9146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:56.661389 9146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:56.662557 9146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:56.662634 9146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:56.662659 9146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:56.662690 9146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:56.662722 9146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:56.664299 9146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:56.664589 9146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:56.664832 9146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:56.670176 9154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:56.670147 9151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:56.670166 9152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:56.670570 9146 server_base.cc:1034] running on GCE node
I20260925 19:11:56.670799 9146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:56.671070 9146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:56.672240 9146 hybrid_clock.cc:648] HybridClock initialized: now 1790363516672227 us; error 32 us; skew 500 ppm
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:56.678740 9146 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:11:56.679252 9146 webserver.cc:460] Webserver started at http://127.27.174.126:39421/ using document root <none> and password file <none>
I20260925 19:11:56.679510 9146 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:56.679586 9146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:56.679764 9146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:11:56.680852 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "90b1f906cfce4a92ae6079b47a11c661"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "7749c1b2a5f7413ef25b1061880778f8"
server_key_iv: "ce2ae8ea9361780e478bcb2a3ca87846"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.681233 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "90b1f906cfce4a92ae6079b47a11c661"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "7749c1b2a5f7413ef25b1061880778f8"
server_key_iv: "ce2ae8ea9361780e478bcb2a3ca87846"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.682708 9146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:11:56.683665 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:56.683957 9146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:56.684077 9146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "90b1f906cfce4a92ae6079b47a11c661"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "7749c1b2a5f7413ef25b1061880778f8"
server_key_iv: "ce2ae8ea9361780e478bcb2a3ca87846"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.684167 9146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:56.701854 9146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:56.704342 9146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:56.704519 9146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:56.708915 9146 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34637
I20260925 19:11:56.708958 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34637 every 8 connection(s)
I20260925 19:11:56.709312 9146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:11:56.710245 9214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:56.712116 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:56.714531 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9146
I20260925 19:11:56.714649 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:11:56.714836 28345 external_mini_cluster.cc:1428] Setting key 5d63eb988fdd6b14d8713a4ba22d52d2
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.125@KRBTEST.COM; defaulting to no policy
I20260925 19:11:56.723662 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.712900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34637 (local address 127.0.0.1:46544)
0925 19:11:56.713092 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.713140 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:56.713614 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:56.713909 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:56.713912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:56.714169 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:56.714343 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.714348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.715914 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.715915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.716613 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.716615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.717330 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.717331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.717392 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.717987 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:56.717996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:56.719771 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:56.721962 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.721969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.721971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.722170 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.722469 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.722470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.722470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.722549 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.723008 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:56.723013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:56.723071 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:11:56.723207 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 19:11:56.723463 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":113,"thread_start_us":75,"threads_started":1}
I20260925 19:11:56.723665 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.713061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.27.174.126:34637)
0925 19:11:56.713208 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:56.713212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:56.713240 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:11:56.713753 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:56.713755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:56.713772 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:56.713851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:56.714485 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.715812 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.716709 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.717224 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.717337 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.719883 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:56.719891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:56.719892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:56.719899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:56.721858 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.722268 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.722272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.722273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.722389 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.722627 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.722628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.722628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.722933 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:56.723019 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:56.723251 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 19:11:56.723402 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260925 19:11:56.724611 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:56.725003 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.724888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39512)
0925 19:11:56.724937 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.724980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:11:56.725183 9162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
I20260925 19:11:56.725710 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.725593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39524)
0925 19:11:56.725657 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.725687 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.125@KRBTEST.COM" created.
W20260925 19:11:56.726029 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:41131: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
I20260925 19:11:56.744046 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:41131
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:11:56.775262 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } attempt: 1
I20260925 19:11:56.775852 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.775717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39536)
0925 19:11:56.775782 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.775824 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:11:56.776227 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.776123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39538)
0925 19:11:56.776174 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.776202 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:11:56.776312 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:41131: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
W20260925 19:11:56.825376 9221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:56.825585 9221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:56.825608 9221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:56.826782 9221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:56.826826 9221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:56.826840 9221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:56.826853 9221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:56.826920 9221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:56.828454 9221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:56.828729 9221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:56.828979 9221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:56.833920 9226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:56.833992 9229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:56.834172 9221 server_base.cc:1034] running on GCE node
W20260925 19:11:56.834259 9227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:56.834590 9221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:56.834877 9221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:56.836040 9221 hybrid_clock.cc:648] HybridClock initialized: now 1790363516836027 us; error 28 us; skew 500 ppm
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:56.838476 9221 init.cc:377] Logged in from keytab as kudu/127.27.174.125@KRBTEST.COM (short username kudu)
I20260925 19:11:56.838893 9221 webserver.cc:460] Webserver started at http://127.27.174.125:34639/ using document root <none> and password file <none>
I20260925 19:11:56.839126 9221 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:56.839172 9221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:56.839325 9221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:11:56.840559 9221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data/instance:
uuid: "d35248859abf4e6ca8d5532fd42987de"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "54bf8ce7b86c21568c44d421156ca98d"
server_key_iv: "2c2d6cfcd65ddbe349c51d4885439c72"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.840926 9221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal/instance:
uuid: "d35248859abf4e6ca8d5532fd42987de"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "54bf8ce7b86c21568c44d421156ca98d"
server_key_iv: "2c2d6cfcd65ddbe349c51d4885439c72"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.842293 9221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:11:56.843189 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:56.843499 9221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:11:56.843575 9221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "d35248859abf4e6ca8d5532fd42987de"
format_stamp: "Formatted at 2026-09-25 19:11:56 on dist-test-slave-fcdn"
server_key: "54bf8ce7b86c21568c44d421156ca98d"
server_key_iv: "2c2d6cfcd65ddbe349c51d4885439c72"
server_key_version: "encryptionkey@0"
I20260925 19:11:56.843693 9221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:56.844457 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } attempt: 2
I20260925 19:11:56.844856 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.844725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39540)
0925 19:11:56.844789 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.844828 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:11:56.845268 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.845181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39544)
0925 19:11:56.845227 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.845251 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:11:56.845397 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:41131: Network error: Client connection negotiation failed: client connection to 127.27.174.125:41131: connect: Connection refused (error 111)
I20260925 19:11:56.863318 9221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:56.865286 9221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:56.865399 9221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:56.869589 9221 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:41131
I20260925 19:11:56.869654 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:41131 every 8 connection(s)
I20260925 19:11:56.870016 9221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
I20260925 19:11:56.870877 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:56.872503 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:56.878770 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9221
I20260925 19:11:56.878881 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-1/wal/instance
I20260925 19:11:56.879184 28345 external_mini_cluster.cc:1428] Setting key 7e95a6cd92460b7ca66efe0b3f4683a7
I20260925 19:11:56.882206 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.873136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34637 (local address 127.0.0.1:46548)
0925 19:11:56.873365 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.873407 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:56.873895 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:56.874163 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:56.874166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:56.874415 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:56.874602 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.874606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.875308 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.875309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.876175 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.876178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.876593 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.876594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.876722 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.877189 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:56.877197 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:56.878486 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:56.880598 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.880607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.880609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.880961 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.881173 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.881173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.881174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.881316 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.881592 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:56.881600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:56.881789 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 19:11:56.881926 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 19:11:56.882049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260925 19:11:56.882211 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.873230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.27.174.126:34637)
0925 19:11:56.873324 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:56.873326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:56.873503 (+ 177us) server_negotiation.cc:407] Connection header received
0925 19:11:56.874015 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:56.874017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:56.874031 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:56.874086 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:56.874707 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.875207 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.876274 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.876492 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.876586 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.878587 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:56.878589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:56.878590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:56.878598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:56.880490 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.881046 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.881047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.881047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.881102 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.881395 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.881395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.881395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.881519 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:56.881570 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:56.881976 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 19:11:56.882070 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 19:11:56.882886 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.124@KRBTEST.COM; defaulting to no policy
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.124@KRBTEST.COM" created.
I20260925 19:11:56.891208 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.883169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39546)
0925 19:11:56.883211 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.883221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:56.883260 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:56.883532 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:56.883533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:56.883707 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:56.883759 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.883760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.885157 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.885157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.885650 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.885651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.886306 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.886307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.886391 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.886824 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:56.886831 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:56.888116 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:56.890001 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.890002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.890002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.890212 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.890426 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.890426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.890427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.890486 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.890857 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:56.890858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:56.890919 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 19:11:56.890983 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:11:56.891117 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 19:11:56.891486 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.883203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.27.174.125:41131)
0925 19:11:56.883355 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:56.883360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:56.883388 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:11:56.883412 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:56.883415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:56.883430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:56.883523 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:56.883841 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.885057 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.885742 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.886224 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.886307 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.888194 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:56.888201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:56.888201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:56.888207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:56.889700 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.890296 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.890299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.890299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.890352 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.890549 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.890550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.890550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.890744 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:56.890842 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:56.891050 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 19:11:56.891389 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":242,"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260925 19:11:56.892315 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:56.892725 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.892573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39618)
0925 19:11:56.892635 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.892695 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 19:11:56.892881 9240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
I20260925 19:11:56.893339 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.893226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39620)
0925 19:11:56.893281 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.893313 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:11:56.893500 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:34259: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
I20260925 19:11:56.908211 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:34259
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:11:56.936594 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } attempt: 3
Sep 25 19:11:56 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:11:56.942747 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } attempt: 1
I20260925 19:11:56.943168 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.943021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39636)
0925 19:11:56.943086 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.943124 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 19:11:56.943594 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.943476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39652)
0925 19:11:56.943534 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.943565 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 19:11:56.943794 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:34259: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
I20260925 19:11:56.944526 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.937004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39552)
0925 19:11:56.937078 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.937094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:56.937161 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:56.937318 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:56.937319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:56.937505 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:56.937583 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.937585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.938256 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.938257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.939070 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:56.939074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.939460 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.939461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:56.939536 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.940018 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:56.940023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:56.941448 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:56.943307 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.943309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.943309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.943554 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.943838 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:56.943839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:56.943839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:56.943900 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:56.944197 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:56.944198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:56.944246 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 19:11:56.944323 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:11:56.944429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 19:11:56.944525 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.27.174.125:41131)
0925 19:11:56.937140 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:56.937142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:56.937159 (+ 17us) server_negotiation.cc:407] Connection header received
0925 19:11:56.937225 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:56.937226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:56.937237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:56.937287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:56.937660 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.938171 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.939167 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:56.939379 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:56.939458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:56.941556 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:56.941557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:56.941557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:56.941565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:56.943180 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.943693 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.943695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.943695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.943757 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:56.943997 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:56.943998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:56.943998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:56.944119 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:56.944193 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:56.944360 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:11:56.944435 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:11:56.945011 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:56.945323 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.945237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39660)
0925 19:11:56.945274 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.945303 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:11:56.945638 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.945585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39666)
0925 19:11:56.945606 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.945626 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 19:11:56.945749 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:34259: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
I20260925 19:11:56.976886 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } attempt: 1
I20260925 19:11:56.977419 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.977282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39670)
0925 19:11:56.977350 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.977391 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:11:56.977823 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:56.977735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39682)
0925 19:11:56.977781 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:56.977806 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:11:56.977955 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:34259: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
W20260925 19:11:56.992383 9296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:56.992584 9296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:56.992606 9296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:56.993661 9296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:56.993698 9296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:56.993712 9296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:56.993724 9296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:56.993736 9296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:56.995118 9296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34637,127.27.174.125:41131,127.27.174.124:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:56.995306 9296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:56.995477 9296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:57.000379 9301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:57.000385 9302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:57.000520 9296 server_base.cc:1034] running on GCE node
W20260925 19:11:57.000461 9304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:57.000880 9296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:57.001165 9296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:57.002338 9296 hybrid_clock.cc:648] HybridClock initialized: now 1790363517002323 us; error 34 us; skew 500 ppm
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:57.005139 9296 init.cc:377] Logged in from keytab as kudu/127.27.174.124@KRBTEST.COM (short username kudu)
I20260925 19:11:57.005530 9296 webserver.cc:460] Webserver started at http://127.27.174.124:35383/ using document root <none> and password file <none>
I20260925 19:11:57.005709 9296 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:57.005748 9296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:57.005836 9296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:11:57.006855 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data/instance:
uuid: "f65abc42c9194f78b16f2621a993f67c"
format_stamp: "Formatted at 2026-09-25 19:11:57 on dist-test-slave-fcdn"
server_key: "ed4d3f853ab5644c3f9f4d76789665ca"
server_key_iv: "97b6998959faaed695051ef5258c505f"
server_key_version: "encryptionkey@0"
I20260925 19:11:57.007169 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal/instance:
uuid: "f65abc42c9194f78b16f2621a993f67c"
format_stamp: "Formatted at 2026-09-25 19:11:57 on dist-test-slave-fcdn"
server_key: "ed4d3f853ab5644c3f9f4d76789665ca"
server_key_iv: "97b6998959faaed695051ef5258c505f"
server_key_version: "encryptionkey@0"
I20260925 19:11:57.008628 9296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:11:57.009477 9311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:57.009742 9296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:11:57.009867 9296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "f65abc42c9194f78b16f2621a993f67c"
format_stamp: "Formatted at 2026-09-25 19:11:57 on dist-test-slave-fcdn"
server_key: "ed4d3f853ab5644c3f9f4d76789665ca"
server_key_iv: "97b6998959faaed695051ef5258c505f"
server_key_version: "encryptionkey@0"
I20260925 19:11:57.009935 9296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:57.011931 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } attempt: 2
I20260925 19:11:57.012463 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.012292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39686)
0925 19:11:57.012387 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.012432 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 19:11:57.012962 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.012836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39702)
0925 19:11:57.012909 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.012940 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 19:11:57.013108 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:34259: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34259: connect: Connection refused (error 111)
I20260925 19:11:57.030134 9296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:57.032681 9296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:57.032853 9296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:57.037115 9296 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:34259
I20260925 19:11:57.037148 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:34259 every 8 connection(s)
I20260925 19:11:57.037520 9296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
I20260925 19:11:57.038450 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:57.040149 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:57.043309 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9296
I20260925 19:11:57.043424 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363366557183-28345-0/minicluster-data/master-2/wal/instance
I20260925 19:11:57.043640 28345 external_mini_cluster.cc:1428] Setting key c76715af109f4e6615b5675c52bc4fe0
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:11:57.050961 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.040875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.27.174.126:34637)
0925 19:11:57.040969 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.040972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.041200 (+ 228us) server_negotiation.cc:407] Connection header received
0925 19:11:57.041728 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.041730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.041742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.041819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.042439 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.042930 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.044378 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.044601 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.044692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.047056 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.047058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.047058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.047066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.049440 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.049900 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.049901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.049901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.049976 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.050241 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.050242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.050243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.050364 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.050440 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.050773 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.050858 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 19:11:57.051180 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.040887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34637 (local address 127.0.0.1:46564)
0925 19:11:57.041070 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.041106 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:57.041603 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:57.041900 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:57.041904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:57.042147 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:57.042335 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.042339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.043027 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.043028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.044278 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.044281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.044700 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.044701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.044881 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.045511 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:57.045520 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:57.046965 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:57.049555 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.049563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.049565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.049813 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.050050 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.050050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.050051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.050167 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.050476 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:57.050482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:57.050618 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 19:11:57.050742 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 19:11:57.050898 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260925 19:11:57.051754 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:57.053125 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } attempt: 2
I20260925 19:11:57.053439 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.044025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.27.174.124:34259)
0925 19:11:57.044202 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.044208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.044233 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:11:57.044262 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.044264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.044285 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.044379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.045095 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.045787 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.046650 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.047109 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.047158 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.049943 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.049950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.049950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.049956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.051996 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.052380 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.052383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.052384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.052437 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.052668 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.052669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.052669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.052807 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.052981 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.053076 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.053153 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:57.060834 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.052112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.27.174.125:41131)
0925 19:11:57.052162 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.052164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.052437 (+ 273us) server_negotiation.cc:407] Connection header received
0925 19:11:57.052527 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.052528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.052538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.052587 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.052832 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.053439 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.054634 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.054879 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.054940 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.056838 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.056840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.056840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.056855 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.059111 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.059585 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.059586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.059586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.059667 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.059891 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.059892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.059893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.060051 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.060122 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.060649 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.060731 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:11:57.061182 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.052031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:41131 (local address 127.0.0.1:39562)
0925 19:11:57.052378 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.052388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:57.052474 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:57.052587 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:57.052588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:57.052716 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:57.052748 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.052749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.053866 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.053866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.054552 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.054554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.054990 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.054991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.055051 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.055451 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:57.055456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:57.056747 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:57.059215 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.059216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.059217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.059493 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.059748 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.059749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.059750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.059806 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.060129 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:57.060130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:57.060205 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 19:11:57.060605 (+ 400us) client_negotiation.cc:240] Negotiation successful
0925 19:11:57.060730 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260925 19:11:57.061203 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.053349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39730)
0925 19:11:57.053553 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.053565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:57.053621 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:57.053799 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:57.053800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:57.053982 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:57.054023 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.054030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.054896 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.054897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.055590 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.055592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.056072 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.056073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.056130 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.056523 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:57.056528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:57.057924 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:57.060009 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.060010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.060010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.060236 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.060462 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.060463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.060464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.060521 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.060816 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:57.060816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:57.060868 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 19:11:57.060941 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:11:57.061106 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260925 19:11:57.061245 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.053415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.27.174.124:34259)
0925 19:11:57.053524 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.053527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.053630 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:11:57.053658 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.053658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.053670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.053757 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.054114 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.054831 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.055696 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.055987 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.056263 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.058023 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.058024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.058025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.058032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.059908 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.060322 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.060322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.060323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.060378 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.060622 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.060623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.060623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.060754 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.060824 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.061048 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.061122 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":98,"threads_started":1}
I20260925 19:11:57.061802 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:11:57.062963 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.054210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.27.174.126:34637)
0925 19:11:57.054240 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.054242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.054249 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:11:57.054267 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.054268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.054275 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.054327 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.054853 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.055498 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.056959 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.057128 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.057213 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.059774 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.059776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.059776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.059783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.061822 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.062213 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.062214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.062214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.062263 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.062487 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.062488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.062488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.062582 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.062707 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.062790 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.062874 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:11:57.063418 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: Bootstrap starting.
I20260925 19:11:57.064268 9214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:57.064613 9214 log.cc:824] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:57.065505 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: No bootstrap required, opened a new log
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:11:57.067791 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.067930 9214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:57.067975 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1f906cfce4a92ae6079b47a11c661, State: Initialized, Role: FOLLOWER
I20260925 19:11:57.068127 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.068405 9371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.068475 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:57.068493 9371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
W20260925 19:11:57.070559 9382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:11:57.070613 9382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:11:57.071497 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.063714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.27.174.124:34259)
0925 19:11:57.063749 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.063751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.063758 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:11:57.063814 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.063815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.063823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.063878 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.064111 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.064562 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.065424 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.065695 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.065790 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.067982 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.067983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.067983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.067989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.070205 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.070607 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.070608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.070608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.070661 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.070863 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.070864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.070864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.071041 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.071131 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.071271 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.071394 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":20}
I20260925 19:11:57.071673 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.062066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39736)
0925 19:11:57.063715 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.063724 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:57.063759 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:57.063879 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:57.063879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:57.064010 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:57.064038 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.064039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.064659 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.064661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.065317 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.065320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.065788 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.065789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.065837 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.066294 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:57.066299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:57.067842 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:57.070302 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.070303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.070303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.070502 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.070752 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.070753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.070753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.070811 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.071123 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:57.071124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:57.071165 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 19:11:57.071232 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:11:57.071344 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1636}
I20260925 19:11:57.072793 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.063554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.27.174.125:41131)
0925 19:11:57.063647 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.063649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.063657 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:11:57.063678 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.063679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.063689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.063741 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.064465 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.065074 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.066253 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.066473 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.066562 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.069344 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.069345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.069346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.069353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.071241 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.071700 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.071702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.071702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.071757 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.071952 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.071953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.071953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.072077 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.072244 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.072630 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.072707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 19:11:57.073217 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: Bootstrap starting.
I20260925 19:11:57.074201 9364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:57.074501 9364 log.cc:824] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:57.075372 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: No bootstrap required, opened a new log
I20260925 19:11:57.072396 9146 master_runner.cc:186] Error getting master registration for 127.27.174.125:41131: OK, instance_id { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" instance_seqno: 1790363516868372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:11:57.075564 9146 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:11:57.076921 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:11:57.077642 9364 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.077806 9364 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:57.077839 9364 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65abc42c9194f78b16f2621a993f67c, State: Initialized, Role: FOLLOWER
I20260925 19:11:57.077966 9364 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.078225 9389 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.078281 9364 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:57.078333 9389 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:11:57.080664 9403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:11:57.080719 9403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:11:57.081667 9296 master_runner.cc:186] Error getting master registration for 127.27.174.125:41131: OK, instance_id { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" instance_seqno: 1790363516868372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:11:57.084869 9296 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:11:57.087284 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.078601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.27.174.126:34637)
0925 19:11:57.078656 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.078659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.079442 (+ 783us) server_negotiation.cc:407] Connection header received
0925 19:11:57.079559 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.079561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.079573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.079652 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.080612 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.081332 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.082392 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.082718 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.082767 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.084317 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.084318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.084319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.084326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.085876 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.086370 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.086371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.086372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.086429 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.086678 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.086679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.086679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.086805 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.086877 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.087090 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.087164 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:11:57.093031 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.078943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.27.174.124:34259)
0925 19:11:57.078982 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.078983 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.079989 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 19:11:57.080061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.080062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.080073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.080103 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.086557 (+ 6454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.087314 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.088361 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.088579 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.088673 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.090077 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.090079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.090079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.090087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.091659 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.092120 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.092122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.092122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.092181 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.092420 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.092421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.092422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.092550 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.092652 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.092846 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.092932 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:11:57.098297 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.079308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.27.174.125:41131)
0925 19:11:57.079344 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.079346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.086282 (+ 6936us) server_negotiation.cc:407] Connection header received
0925 19:11:57.086353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.086354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.086366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.086430 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.092350 (+ 5920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.093083 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.094035 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.094270 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.094306 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.095646 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.095648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.095648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.095656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.097113 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.097482 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.097483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.097484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.097545 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.097750 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.097751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.097752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.097866 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.097974 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.098087 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.098164 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:11:57.104302 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } attempt: 3
Sep 25 19:11:57 dist-test-slave-fcdn krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:11:57.111408 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.104590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34259 (local address 127.0.0.1:39756)
0925 19:11:57.104642 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:11:57.104654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:11:57.104693 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:11:57.104831 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:11:57.104832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:11:57.104965 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:11:57.105000 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.105001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.105620 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.105621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.106109 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:11:57.106110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.106483 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.106484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:11:57.106540 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.106977 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:11:57.106982 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:11:57.108375 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:11:57.110280 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.110280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.110281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.110502 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.110711 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:11:57.110712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:11:57.110712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:11:57.110773 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:11:57.111064 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:11:57.111065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:11:57.111109 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:11:57.111206 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 19:11:57.111310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:11:57.111408 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:57.104645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.27.174.124:34259)
0925 19:11:57.104687 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:57.104689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:57.104697 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:11:57.104748 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:57.104748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:57.104765 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:57.104815 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:57.105078 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.105539 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.106195 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:57.106395 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:57.106482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:57.108476 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:57.108477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:57.108478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:57.108485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:57.110191 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.110581 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.110582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.110582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.110642 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:57.110844 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:57.110845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:57.110845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:57.111001 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:57.111060 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:57.111233 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:11:57.111318 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:11:57.113420 9289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: Bootstrap starting.
I20260925 19:11:57.114187 9289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:57.114517 9289 log.cc:824] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:57.115357 9289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: No bootstrap required, opened a new log
I20260925 19:11:57.117568 9289 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.117758 9289 raft_consensus.cc:374] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:57.117807 9289 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35248859abf4e6ca8d5532fd42987de, State: Initialized, Role: FOLLOWER
I20260925 19:11:57.117938 9289 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.118229 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.118302 9289 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:57.118362 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:11:57.120203 9417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:11:57.120332 9417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:11:57.122252 9221 master_runner.cc:263] No leader master found from master d35248859abf4e6ca8d5532fd42987de
I20260925 19:11:57.122306 9221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:11:57.198993 9406 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:11:57.199115 9406 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.199364 9406 leader_election.cc:288] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90b1f906cfce4a92ae6079b47a11c661 (127.27.174.126:34637), f65abc42c9194f78b16f2621a993f67c (127.27.174.124:34259)
I20260925 19:11:57.200067 9338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d35248859abf4e6ca8d5532fd42987de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65abc42c9194f78b16f2621a993f67c" is_pre_election: true
I20260925 19:11:57.200248 9338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35248859abf4e6ca8d5532fd42987de in term 0.
I20260925 19:11:57.200677 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d35248859abf4e6ca8d5532fd42987de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1f906cfce4a92ae6079b47a11c661" is_pre_election: true
I20260925 19:11:57.200862 9188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35248859abf4e6ca8d5532fd42987de in term 0.
I20260925 19:11:57.201164 9240 leader_election.cc:302] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b1f906cfce4a92ae6079b47a11c661, d35248859abf4e6ca8d5532fd42987de; no voters:
I20260925 19:11:57.201399 9406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:11:57.201462 9406 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:11:57.201486 9406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:57.202456 9406 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.202581 9406 leader_election.cc:288] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [CANDIDATE]: Term 1 election: Requested vote from peers 90b1f906cfce4a92ae6079b47a11c661 (127.27.174.126:34637), f65abc42c9194f78b16f2621a993f67c (127.27.174.124:34259)
I20260925 19:11:57.202919 9338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d35248859abf4e6ca8d5532fd42987de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65abc42c9194f78b16f2621a993f67c"
I20260925 19:11:57.203008 9338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:57.203792 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d35248859abf4e6ca8d5532fd42987de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1f906cfce4a92ae6079b47a11c661"
I20260925 19:11:57.203888 9188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:57.204773 9188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35248859abf4e6ca8d5532fd42987de in term 1.
I20260925 19:11:57.205304 9338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35248859abf4e6ca8d5532fd42987de in term 1.
W20260925 19:11:57.205319 9167 tablet.cc:2369] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:11:57.205595 9240 leader_election.cc:302] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b1f906cfce4a92ae6079b47a11c661, d35248859abf4e6ca8d5532fd42987de; no voters:
I20260925 19:11:57.205796 9406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:57.205976 9406 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [term 1 LEADER]: Becoming Leader. State: Replica: d35248859abf4e6ca8d5532fd42987de, State: Running, Role: LEADER
I20260925 19:11:57.206068 9406 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } }
I20260925 19:11:57.206486 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35248859abf4e6ca8d5532fd42987de. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.206563 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:57.206948 9421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:11:57.207170 9421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:11:57.210304 9338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [term 1 FOLLOWER]: Refusing update from remote peer d35248859abf4e6ca8d5532fd42987de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:57.210700 9406 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:57.212158 9389 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35248859abf4e6ca8d5532fd42987de. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.212311 9389 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:11:57.212822 9188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [term 1 FOLLOWER]: Refusing update from remote peer d35248859abf4e6ca8d5532fd42987de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:11:57.213220 9406 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:11:57.214537 9371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35248859abf4e6ca8d5532fd42987de. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.214627 9371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:11:57.215714 9419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.215821 9419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:57.216266 9371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.216351 9371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:11:57.216632 9422 mvcc.cc:205] Tried to move back new op lower bound from 7333328966490959872 to 7333328966476754944. Current Snapshot: MvccSnapshot[applied={T|T < 7333328966490959872}]
I20260925 19:11:57.217525 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.217602 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:57.218078 9389 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d35248859abf4e6ca8d5532fd42987de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b1f906cfce4a92ae6079b47a11c661" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34637 } } peers { permanent_uuid: "d35248859abf4e6ca8d5532fd42987de" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 41131 } } peers { permanent_uuid: "f65abc42c9194f78b16f2621a993f67c" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34259 } } }
I20260925 19:11:57.218170 9389 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:11:57.219003 9423 mvcc.cc:205] Tried to move back new op lower bound from 7333328966490959872 to 7333328966476754944. Current Snapshot: MvccSnapshot[applied={T|T < 7333328966490959872}]
I20260925 19:11:57.219658 9421 catalog_manager.cc:1347] Generated new cluster ID: 0a623f724c84449791471ec0a85cfe92
I20260925 19:11:57.219720 9421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:11:57.224058 9425 mvcc.cc:205] Tried to move back new op lower bound from 7333328966490959872 to 7333328966476754944. Current Snapshot: MvccSnapshot[applied={T|T < 7333328966490959872}]
I20260925 19:11:57.243436 9421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:11:57.244319 9421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:11:57.251259 9421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d35248859abf4e6ca8d5532fd42987de: Generated new TSK 0
I20260925 19:11:57.251530 9421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:11:58.071036 9382 catalog_manager.cc:1259] Loaded cluster ID: 0a623f724c84449791471ec0a85cfe92
I20260925 19:11:58.071105 9382 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: loading cluster ID for follower catalog manager: success
I20260925 19:11:58.071895 9382 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: acquiring CA information for follower catalog manager: success
I20260925 19:11:58.072186 9382 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90b1f906cfce4a92ae6079b47a11c661: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:11:58.081135 9403 catalog_manager.cc:1259] Loaded cluster ID: 0a623f724c84449791471ec0a85cfe92
I20260925 19:11:58.081207 9403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: loading cluster ID for follower catalog manager: success
I20260925 19:11:58.081988 9403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: acquiring CA information for follower catalog manager: success
I20260925 19:11:58.082240 9403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f65abc42c9194f78b16f2621a993f67c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:11:58.311177 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9146
I20260925 19:11:58.316663 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9221
I20260925 19:11:58.322736 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9296
2026-09-25T19:11:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2931 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:11:58 dist-test-slave-fcdn krb5kdc[9435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:11:58 dist-test-slave-fcdn krb5kdc[9435](info): set up 2 sockets
Sep 25 19:11:58 dist-test-slave-fcdn krb5kdc[9435](info): commencing operation
krb5kdc: starting...
W20260925 19:11:59.379736 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:11:59.499477 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40291
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:11:59.579494 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:11:59.579707 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:11:59.579730 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:11:59.580909 9448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:11:59.580952 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:11:59.580966 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:11:59.580979 9448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:11:59.581041 9448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:11:59.582472 9448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:11:59.582731 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:11:59.582978 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:11:59.587917 9453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:11:59.587910 9454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:59.587952 9448 server_base.cc:1034] running on GCE node
W20260925 19:11:59.587917 9456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:11:59.588414 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:11:59.588707 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:11:59.589869 9448 hybrid_clock.cc:648] HybridClock initialized: now 1790363519589854 us; error 29 us; skew 500 ppm
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:11:59.592397 9448 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:11:59.592852 9448 webserver.cc:460] Webserver started at http://127.27.174.126:42661/ using document root <none> and password file <none>
I20260925 19:11:59.593073 9448 fs_manager.cc:359] Metadata directory not provided
I20260925 19:11:59.593117 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:11:59.593259 9448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:11:59.594322 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "527fcfede8de4bdaae2e613b50bfb747"
format_stamp: "Formatted at 2026-09-25 19:11:59 on dist-test-slave-fcdn"
server_key: "9c35c447cbc0fbdcc3bee5a29b15c420"
server_key_iv: "492fb063234ed6abfb39c065cf3dc4ce"
server_key_version: "encryptionkey@0"
I20260925 19:11:59.594616 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "527fcfede8de4bdaae2e613b50bfb747"
format_stamp: "Formatted at 2026-09-25 19:11:59 on dist-test-slave-fcdn"
server_key: "9c35c447cbc0fbdcc3bee5a29b15c420"
server_key_iv: "492fb063234ed6abfb39c065cf3dc4ce"
server_key_version: "encryptionkey@0"
I20260925 19:11:59.595917 9448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:11:59.596709 9463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:11:59.596916 9448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:11:59.597012 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "527fcfede8de4bdaae2e613b50bfb747"
format_stamp: "Formatted at 2026-09-25 19:11:59 on dist-test-slave-fcdn"
server_key: "9c35c447cbc0fbdcc3bee5a29b15c420"
server_key_iv: "492fb063234ed6abfb39c065cf3dc4ce"
server_key_version: "encryptionkey@0"
I20260925 19:11:59.597087 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:11:59.616820 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:11:59.619206 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:11:59.619320 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:11:59.623317 9448 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:40291
I20260925 19:11:59.623355 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:40291 every 8 connection(s)
I20260925 19:11:59.623792 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:11:59.624722 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:11:59.626899 9516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747: Bootstrap starting.
I20260925 19:11:59.627426 9516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747: Neither blocks nor log segments found. Creating new log.
I20260925 19:11:59.627672 9516 log.cc:824] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747: Log is configured to *not* fsync() on all Append() calls
I20260925 19:11:59.628300 9516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747: No bootstrap required, opened a new log
I20260925 19:11:59.629813 9516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } }
I20260925 19:11:59.629945 9516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:11:59.629967 9516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527fcfede8de4bdaae2e613b50bfb747, State: Initialized, Role: FOLLOWER
I20260925 19:11:59.630115 9516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } }
I20260925 19:11:59.630200 9516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:11:59.630225 9516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:11:59.630254 9516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:11:59.630738 9516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } }
I20260925 19:11:59.630827 9516 leader_election.cc:302] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527fcfede8de4bdaae2e613b50bfb747; no voters:
I20260925 19:11:59.630951 9516 leader_election.cc:288] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:11:59.631035 9519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:11:59.631211 9519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [term 1 LEADER]: Becoming Leader. State: Replica: 527fcfede8de4bdaae2e613b50bfb747, State: Running, Role: LEADER
I20260925 19:11:59.631237 9516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:11:59.631312 9519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } }
I20260925 19:11:59.631712 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527fcfede8de4bdaae2e613b50bfb747. Latest consensus state: current_term: 1 leader_uuid: "527fcfede8de4bdaae2e613b50bfb747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } } }
I20260925 19:11:59.631844 9520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527fcfede8de4bdaae2e613b50bfb747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527fcfede8de4bdaae2e613b50bfb747" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40291 } } }
I20260925 19:11:59.631930 9520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:59.631901 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747 [sys.catalog]: This master's current role is: LEADER
I20260925 19:11:59.632347 9528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:11:59.632869 9528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:11:59.634018 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9448
I20260925 19:11:59.634117 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:11:59.634296 28345 external_mini_cluster.cc:1428] Setting key b61fee6de1ead1f6e994cf88b13fee0a
I20260925 19:11:59.634419 9528 catalog_manager.cc:1347] Generated new cluster ID: ad708fedfd3745078837370f5851c008
I20260925 19:11:59.634464 9528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:11:59.649964 9528 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:11:59.650660 9528 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:11:59.650756 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:11:59.634716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.27.174.126:40291)
0925 19:11:59.634898 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 19:11:59.634908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:11:59.639869 (+ 4961us) server_negotiation.cc:407] Connection header received
0925 19:11:59.639999 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:11:59.640006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:11:59.640064 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:11:59.640212 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:11:59.640984 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:59.642320 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:59.643424 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:11:59.644072 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:11:59.644168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:11:59.646947 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:11:59.646952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:11:59.646952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:11:59.646965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:11:59.649034 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:59.649450 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:59.649453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:59.649454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:59.649552 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:11:59.649805 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:11:59.649806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:11:59.649807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:11:59.650107 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:11:59.650271 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 19:11:59.650370 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 19:11:59.650511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260925 19:11:59.651682 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:11:59.654131 9528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527fcfede8de4bdaae2e613b50bfb747: Generated new TSK 0
I20260925 19:11:59.654313 9528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:11:59.705793 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9448
2026-09-25T19:11:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1379 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9544](info): set up 2 sockets
Sep 25 19:11:59 dist-test-slave-fcdn krb5kdc[9544](info): commencing operation
krb5kdc: starting...
W20260925 19:12:00.763787 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:12:00 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:12:00.883136 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:00.962994 9557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:00.963199 9557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:00.963222 9557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:00.964418 9557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:12:00.964462 9557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:00.964476 9557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:00.964489 9557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:12:00.964501 9557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:12:00.965898 9557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:00.966085 9557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:00.966253 9557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:00.971220 9562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:00.971252 9565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:00.971274 9557 server_base.cc:1034] running on GCE node
W20260925 19:12:00.971239 9563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:00.971745 9557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:00.972080 9557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:00.973242 9557 hybrid_clock.cc:648] HybridClock initialized: now 1790363520973226 us; error 28 us; skew 500 ppm
Sep 25 19:12:00 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:00.975904 9557 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:12:00.976366 9557 webserver.cc:460] Webserver started at http://127.27.174.126:36077/ using document root <none> and password file <none>
I20260925 19:12:00.976608 9557 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:00.976680 9557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:00.976826 9557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:00.977954 9557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "f77aec91b2104c15933714ce08a119dd"
format_stamp: "Formatted at 2026-09-25 19:12:00 on dist-test-slave-fcdn"
server_key: "1ed0fcf98877420d12cf7de78345e1fd"
server_key_iv: "7257689deff222f02545b0aac59e39ac"
server_key_version: "encryptionkey@0"
I20260925 19:12:00.978325 9557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "f77aec91b2104c15933714ce08a119dd"
format_stamp: "Formatted at 2026-09-25 19:12:00 on dist-test-slave-fcdn"
server_key: "1ed0fcf98877420d12cf7de78345e1fd"
server_key_iv: "7257689deff222f02545b0aac59e39ac"
server_key_version: "encryptionkey@0"
I20260925 19:12:00.979691 9557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:12:00.980453 9572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:00.980640 9557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:00.980741 9557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "f77aec91b2104c15933714ce08a119dd"
format_stamp: "Formatted at 2026-09-25 19:12:00 on dist-test-slave-fcdn"
server_key: "1ed0fcf98877420d12cf7de78345e1fd"
server_key_iv: "7257689deff222f02545b0aac59e39ac"
server_key_version: "encryptionkey@0"
I20260925 19:12:00.980822 9557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.002504 9557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.005113 9557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.005338 9557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.009611 9557 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43315
I20260925 19:12:01.009655 9624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43315 every 8 connection(s)
I20260925 19:12:01.010056 9557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:12:01.011051 9625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:01.013491 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap starting.
I20260925 19:12:01.014108 9625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:01.014407 9625 log.cc:824] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.015194 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: No bootstrap required, opened a new log
I20260925 19:12:01.016937 9625 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.017079 9625 raft_consensus.cc:374] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:01.017102 9625 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.017264 9625 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.017328 9625 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:12:01.017352 9625 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:12:01.017405 9625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:01.018193 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9557
I20260925 19:12:01.018225 9625 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.018319 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:12:01.018340 9625 leader_election.cc:302] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77aec91b2104c15933714ce08a119dd; no voters:
I20260925 19:12:01.018529 9625 leader_election.cc:288] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:12:01.018560 9628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:01.018555 28345 external_mini_cluster.cc:1428] Setting key 34fad6d3a25d682738e557cda96fcbd7
I20260925 19:12:01.018663 9628 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 LEADER]: Becoming Leader. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Running, Role: LEADER
I20260925 19:12:01.018801 9625 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:12:01.018802 9628 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.020255 9630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:01.020252 9629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77aec91b2104c15933714ce08a119dd. Latest consensus state: current_term: 1 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:01.020417 9630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:01.020430 9629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:01.020618 9645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:12:01.021399 9645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:12:01.023272 9645 catalog_manager.cc:1347] Generated new cluster ID: 7ed5827a1cc14484ba1ce97845adb8f1
I20260925 19:12:01.023335 9645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:01.031369 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.019095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.27.174.126:43315)
0925 19:12:01.019305 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.019315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.019358 (+ 43us) server_negotiation.cc:407] Connection header received
0925 19:12:01.019483 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.019514 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.019578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.020826 (+ 1248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.020957 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.022206 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.023278 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.023908 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.024016 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.026887 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.026892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.026892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.026906 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.029614 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.030022 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.030025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.030026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.030153 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.030376 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.030377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.030378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.030726 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.030908 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.030989 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.031137 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260925 19:12:01.038431 9645 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:12:01.039000 9645 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:12:01.045056 9645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Generated new TSK 0
I20260925 19:12:01.045276 9645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:12:01.080036 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:01.163841 9651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:01.164050 9651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:01.164074 9651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:01.165293 9651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:01.165337 9651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:01.165364 9651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:12:01.166769 9651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:01.166961 9651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:01.167140 9651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:01.172266 9656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.172253 9659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.172256 9657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.172274 9651 server_base.cc:1034] running on GCE node
I20260925 19:12:01.172693 9651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:01.172933 9651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:01.174106 9651 hybrid_clock.cc:648] HybridClock initialized: now 1790363521174090 us; error 30 us; skew 500 ppm
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:01.176658 9651 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:12:01.177143 9651 webserver.cc:460] Webserver started at http://127.27.174.65:35287/ using document root <none> and password file <none>
I20260925 19:12:01.177420 9651 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:01.177511 9651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:01.177649 9651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:01.178706 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "50ac90119ef5401384c49cc76a95d242"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "b38a97bf18f67bf9d6759692be073e40"
server_key_iv: "831916b22bb791e735d9e427e21c643f"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.179097 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "50ac90119ef5401384c49cc76a95d242"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "b38a97bf18f67bf9d6759692be073e40"
server_key_iv: "831916b22bb791e735d9e427e21c643f"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.180594 9651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:12:01.181413 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.181628 9651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:01.181743 9651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "50ac90119ef5401384c49cc76a95d242"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "b38a97bf18f67bf9d6759692be073e40"
server_key_iv: "831916b22bb791e735d9e427e21c643f"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.181828 9651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.214073 9651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.216545 9651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.216734 9651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.217001 9651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:01.217330 9651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:01.217427 9651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.217494 9651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:01.217515 9651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.223743 9651 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37643
I20260925 19:12:01.223809 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37643 every 8 connection(s)
I20260925 19:12:01.224121 9651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:12:01.225405 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9651
I20260925 19:12:01.225520 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:12:01.225692 28345 external_mini_cluster.cc:1428] Setting key 99a0bd9532dc51d3fc5fbcb8942d146a
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:12:01.235127 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.224817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:53825 (local address 127.27.174.126:43315)
0925 19:12:01.224915 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.224917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.225181 (+ 264us) server_negotiation.cc:407] Connection header received
0925 19:12:01.225951 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.225953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.225968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.226093 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.227220 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.228013 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.229059 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.229340 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.229433 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.232046 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.232048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.232049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.232057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.233631 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.234070 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.234071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.234071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.234129 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.234438 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.234439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.234439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.234567 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.234648 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.234947 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.235034 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 19:12:01.235479 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.224830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:53825)
0925 19:12:01.225011 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.225049 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.225764 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.226202 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.226206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.226899 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:01.227095 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.227100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.228116 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.228117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.228946 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.228948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.229453 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.229454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.229601 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.230216 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:01.230225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:01.231929 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:01.233736 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.233743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.233746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.233973 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.234199 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.234199 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.234200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.234360 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.234639 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:01.234646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:01.234801 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.234925 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.235090 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260925 19:12:01.236078 9780 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:01.236168 9780 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:01.236385 9780 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:01.237000 9589 ts_manager.cc:194] Registered new tserver with Master: 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20260925 19:12:01.238133 9589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:53825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:12:01.255739 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:01.337318 9785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:01.337520 9785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:01.337543 9785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:01.338706 9785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:01.338748 9785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:01.338774 9785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:12:01.340296 9785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:01.340567 9785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:01.340802 9785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:01.346086 9790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.346109 9791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.346089 9793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.346366 9785 server_base.cc:1034] running on GCE node
I20260925 19:12:01.346547 9785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:01.346728 9785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:01.347940 9785 hybrid_clock.cc:648] HybridClock initialized: now 1790363521347928 us; error 25 us; skew 500 ppm
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:01.350426 9785 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:12:01.350925 9785 webserver.cc:460] Webserver started at http://127.27.174.66:46491/ using document root <none> and password file <none>
I20260925 19:12:01.351210 9785 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:01.351289 9785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:01.351428 9785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:01.352921 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "91986472596b4296b00b2c72ff869011"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "8a835ac9cd5f6062a6ee7e488d0f203f"
server_key_iv: "223c75bac4893b350599f918b892d3a6"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.353336 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "91986472596b4296b00b2c72ff869011"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "8a835ac9cd5f6062a6ee7e488d0f203f"
server_key_iv: "223c75bac4893b350599f918b892d3a6"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.354964 9785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:12:01.355806 9800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.355997 9785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:12:01.356094 9785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "91986472596b4296b00b2c72ff869011"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "8a835ac9cd5f6062a6ee7e488d0f203f"
server_key_iv: "223c75bac4893b350599f918b892d3a6"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.356182 9785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.372298 9785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.374480 9785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.374671 9785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.374924 9785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:01.375247 9785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:01.375306 9785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.375357 9785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:01.375385 9785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.381935 9785 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41689
I20260925 19:12:01.381983 9913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41689 every 8 connection(s)
I20260925 19:12:01.382318 9785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:01.390784 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9785
I20260925 19:12:01.390892 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:12:01.391090 28345 external_mini_cluster.cc:1428] Setting key a0a970e3e7754a488cc45462a7250a15
I20260925 19:12:01.391992 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.383154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:55159 (local address 127.27.174.126:43315)
0925 19:12:01.383231 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.383234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.383406 (+ 172us) server_negotiation.cc:407] Connection header received
0925 19:12:01.384025 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.384026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.384037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.384119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.385141 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.385640 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.386673 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.386869 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.386923 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.389107 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.389108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.389108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.389115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.390417 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.390839 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.390840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.390841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.390927 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.391194 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.391195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.391195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.391325 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.391408 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.391762 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.391888 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:12:01.392235 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.383124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:55159)
0925 19:12:01.383299 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.383334 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.383927 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.384182 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.384186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.384852 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:01.385053 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.385059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.385724 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.385724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.386598 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.386600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.386943 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.386944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.387040 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.387654 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:01.387668 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:01.389019 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:01.390496 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.390502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.390503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.390737 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.391005 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.391005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.391006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.391114 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.391399 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:01.391406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:01.391589 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.391727 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.391948 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260925 19:12:01.392710 9914 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:01.392820 9914 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:01.393000 9914 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:01.393402 9589 ts_manager.cc:194] Registered new tserver with Master: 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
I20260925 19:12:01.393793 9589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:55159
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:12:01.419273 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:01.502453 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:01.502660 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:01.502686 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:01.504248 9919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:01.504303 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:01.504335 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:12:01.505993 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:01.506412 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:01.506695 9919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:01.512162 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.512203 9919 server_base.cc:1034] running on GCE node
W20260925 19:12:01.512311 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.512449 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.512709 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:01.512971 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:01.514155 9919 hybrid_clock.cc:648] HybridClock initialized: now 1790363521514150 us; error 58 us; skew 500 ppm
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:01.516812 9919 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:12:01.517277 9919 webserver.cc:460] Webserver started at http://127.27.174.67:35419/ using document root <none> and password file <none>
I20260925 19:12:01.517472 9919 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:01.517513 9919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:01.517599 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:01.518704 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "2f30deba566e4e1f80d3b2cf8286a993"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "7b539e4e9e1d3f2f14a2423ffdf757b7"
server_key_iv: "ea3211de6bfa32bb801660020c183dd2"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.519071 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "2f30deba566e4e1f80d3b2cf8286a993"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "7b539e4e9e1d3f2f14a2423ffdf757b7"
server_key_iv: "ea3211de6bfa32bb801660020c183dd2"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.520705 9919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:12:01.521449 9934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.521659 9919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:01.521754 9919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2f30deba566e4e1f80d3b2cf8286a993"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "7b539e4e9e1d3f2f14a2423ffdf757b7"
server_key_iv: "ea3211de6bfa32bb801660020c183dd2"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.521836 9919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.535104 9919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.537762 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.537910 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.538175 9919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:01.538503 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:01.538537 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.538559 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:01.538571 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.544574 9919 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42783
I20260925 19:12:01.544628 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42783 every 8 connection(s)
I20260925 19:12:01.545025 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:01.554328 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.545873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35593 (local address 127.27.174.126:43315)
0925 19:12:01.545951 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.545954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.546162 (+ 208us) server_negotiation.cc:407] Connection header received
0925 19:12:01.546641 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.546642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.546652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.546713 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.547737 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.548238 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.549208 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.549395 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.549449 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.551506 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.551507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.551508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.551514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.553166 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.553545 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.553546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.553546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.553598 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.553809 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.553810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.553810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.553912 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.553962 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.554196 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.554263 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:12:01.554535 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.545799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:35593)
0925 19:12:01.546018 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.546062 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.546545 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.546767 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.546771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.547363 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:01.547586 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.547590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.548329 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.548330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.549129 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.549131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.549469 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.549470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.549568 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.550025 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:01.550033 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:01.551386 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:01.553246 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.553253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.553254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.553476 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.553657 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.553658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.553658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.553754 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.553969 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:01.553973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:01.554086 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.554170 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.554287 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260925 19:12:01.554811 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 9919
I20260925 19:12:01.554915 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:12:01.555080 28345 external_mini_cluster.cc:1428] Setting key 5179b464b43715053e886815d7dd7d9d
I20260925 19:12:01.555145 10048 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:01.555290 10048 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:01.555500 10048 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:01.556043 9589 ts_manager.cc:194] Registered new tserver with Master: 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783)
I20260925 19:12:01.556396 9589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:35593
I20260925 19:12:01.567253 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:12:01.576155 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.568891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.27.174.126:43315)
0925 19:12:01.568937 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.568939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.569070 (+ 131us) server_negotiation.cc:407] Connection header received
0925 19:12:01.569151 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.569152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.569162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.569233 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.569816 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.570320 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.571065 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.571254 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.571679 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.573099 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.573101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.573101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.573108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.574730 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.575119 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.575120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.575120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.575179 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.575413 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.575413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.575414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.575569 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.575719 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.575960 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.576049 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:12:01.578136 9589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:12:01.579763 9589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:12:01.586804 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41689 (local address 127.0.0.1:33026)
0925 19:12:01.582735 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.582745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.582813 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.583017 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.583017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.583019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:01.583064 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.583066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584340 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.584341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.585782 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.585785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586480 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.586481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.586559 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.586572 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.586667 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.586725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":59,"thread_start_us":74,"threads_started":1}
I20260925 19:12:01.586815 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.27.174.66:41689)
0925 19:12:01.582723 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.582728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.582872 (+ 144us) server_negotiation.cc:407] Connection header received
0925 19:12:01.582917 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.582919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.582935 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.583011 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:01.583119 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584231 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.585879 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586382 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.586469 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.586530 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.586695 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.586734 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260925 19:12:01.586984 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42783 (local address 127.0.0.1:42398)
0925 19:12:01.582439 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.582451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.582501 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.582808 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.582811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.582813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:01.582895 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.582897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584275 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.584276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.585793 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.585795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586713 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.586714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.586791 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.586795 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.586826 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.586886 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":71,"thread_start_us":84,"threads_started":1}
I20260925 19:12:01.587193 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.0.0.1:57908)
0925 19:12:01.582352 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.582397 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.582463 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.582808 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.582811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.582816 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:01.582895 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.582897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584299 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.584300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.585812 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.585814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586782 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.586783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.586840 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.586842 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.587024 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.587132 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":128,"thread_start_us":56,"threads_started":1}
I20260925 19:12:01.587366 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.27.174.67:42783)
0925 19:12:01.582618 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.582621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.582630 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:12:01.582669 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.582670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.582685 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.583145 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:01.583218 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584053 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.585910 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586610 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.587110 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.587181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.587207 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.587275 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":102,"threads_started":1}
I20260925 19:12:01.587548 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.582223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.27.174.65:37643)
0925 19:12:01.582543 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.582548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.582560 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:12:01.582612 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.582615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.582631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.582714 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:01.582965 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.584201 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.585902 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.586610 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.586684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.586740 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.587350 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.587468 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":89,"threads_started":1}
I20260925 19:12:01.587682 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 926b748dcedc49e8abf249db848b9f8e (DEFAULT_TABLE table=test-table [id=cc56aff518ff40ffb1e2c30eca30e863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:01.588068 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926b748dcedc49e8abf249db848b9f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:01.588045 9982 tablet_service.cc:1467] Processing CreateTablet for tablet 926b748dcedc49e8abf249db848b9f8e (DEFAULT_TABLE table=test-table [id=cc56aff518ff40ffb1e2c30eca30e863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:01.588172 9714 tablet_service.cc:1467] Processing CreateTablet for tablet 926b748dcedc49e8abf249db848b9f8e (DEFAULT_TABLE table=test-table [id=cc56aff518ff40ffb1e2c30eca30e863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:01.588375 9982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926b748dcedc49e8abf249db848b9f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:01.588442 9714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926b748dcedc49e8abf249db848b9f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:01.590689 10065 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Bootstrap starting.
I20260925 19:12:01.591706 10065 tablet_bootstrap.cc:654] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:01.591780 10066 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Bootstrap starting.
I20260925 19:12:01.592428 10065 log.cc:824] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.592614 10066 tablet_bootstrap.cc:654] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:01.593164 10066 log.cc:824] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.593317 10065 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: No bootstrap required, opened a new log
I20260925 19:12:01.593392 10065 ts_tablet_manager.cc:1397] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 19:12:01.591590 10067 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Bootstrap starting.
I20260925 19:12:01.594065 10067 tablet_bootstrap.cc:654] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:01.594615 10067 log.cc:824] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.595367 10067 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: No bootstrap required, opened a new log
I20260925 19:12:01.595440 10067 ts_tablet_manager.cc:1397] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 19:12:01.595702 10065 raft_consensus.cc:348] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.595871 10065 raft_consensus.cc:374] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:01.595903 10065 raft_consensus.cc:729] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91986472596b4296b00b2c72ff869011, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.596052 10065 consensus_queue.cc:260] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.596253 10066 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: No bootstrap required, opened a new log
I20260925 19:12:01.596343 10065 ts_tablet_manager.cc:1428] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:12:01.596379 10066 ts_tablet_manager.cc:1397] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 19:12:01.596467 9914 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
I20260925 19:12:01.597692 10067 raft_consensus.cc:348] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.597817 10067 raft_consensus.cc:374] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:01.597837 10067 raft_consensus.cc:729] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ac90119ef5401384c49cc76a95d242, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.597960 10067 consensus_queue.cc:260] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.598242 10067 ts_tablet_manager.cc:1428] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 19:12:01.598268 9780 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
I20260925 19:12:01.598758 10066 raft_consensus.cc:348] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.598917 10066 raft_consensus.cc:374] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:01.598953 10066 raft_consensus.cc:729] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f30deba566e4e1f80d3b2cf8286a993, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.599097 10066 consensus_queue.cc:260] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.599413 10066 ts_tablet_manager.cc:1428] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:12:01.599421 10048 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
I20260925 19:12:01.623402 10071 raft_consensus.cc:482] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:01.623574 10071 raft_consensus.cc:504] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.624032 10071 leader_election.cc:288] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643), 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783)
I20260925 19:12:01.627046 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.624404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:35023 (local address 127.27.174.67:42783)
0925 19:12:01.624471 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.624473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.624480 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:12:01.624508 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.624509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.624517 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.624577 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:01.624743 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.625231 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626304 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.626688 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626774 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.626783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.626940 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.626977 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:12:01.627046 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.624225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42783 (local address 127.27.174.66:35023)
0925 19:12:01.624272 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.624285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.624356 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.624582 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.624583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.624584 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:01.624668 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.624670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.625330 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.625331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.626197 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.626199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.626772 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.626856 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.626864 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.626932 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.626973 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:12:01.627233 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.624227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.27.174.66:53887)
0925 19:12:01.624514 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.624524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.624612 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.624780 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.624780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.624782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:01.624865 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.624867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.625582 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.625583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.626383 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.626385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.626967 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.627039 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.627043 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.627112 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.627153 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":59,"thread_start_us":164,"threads_started":1}
I20260925 19:12:01.627231 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.624323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53887 (local address 127.27.174.65:37643)
0925 19:12:01.624396 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.624398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.624599 (+ 201us) server_negotiation.cc:407] Connection header received
0925 19:12:01.624667 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.624668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.624681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.624773 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:01.624944 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.625467 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626477 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.626880 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.626966 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.626975 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.627132 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.627171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:12:01.627413 10002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926b748dcedc49e8abf249db848b9f8e" candidate_uuid: "91986472596b4296b00b2c72ff869011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f30deba566e4e1f80d3b2cf8286a993" is_pre_election: true
I20260925 19:12:01.627579 10002 raft_consensus.cc:2454] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91986472596b4296b00b2c72ff869011 in term 0.
I20260925 19:12:01.627581 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926b748dcedc49e8abf249db848b9f8e" candidate_uuid: "91986472596b4296b00b2c72ff869011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ac90119ef5401384c49cc76a95d242" is_pre_election: true
I20260925 19:12:01.627760 9734 raft_consensus.cc:2454] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91986472596b4296b00b2c72ff869011 in term 0.
I20260925 19:12:01.627857 9801 leader_election.cc:302] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011; no voters:
I20260925 19:12:01.628051 10071 raft_consensus.cc:2781] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:12:01.628108 10071 raft_consensus.cc:482] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:12:01.628140 10071 raft_consensus.cc:3037] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:01.628844 10071 raft_consensus.cc:504] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.629000 10071 leader_election.cc:288] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [CANDIDATE]: Term 1 election: Requested vote from peers 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643), 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783)
I20260925 19:12:01.629138 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926b748dcedc49e8abf249db848b9f8e" candidate_uuid: "91986472596b4296b00b2c72ff869011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ac90119ef5401384c49cc76a95d242"
I20260925 19:12:01.629164 10002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926b748dcedc49e8abf249db848b9f8e" candidate_uuid: "91986472596b4296b00b2c72ff869011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f30deba566e4e1f80d3b2cf8286a993"
I20260925 19:12:01.629235 9734 raft_consensus.cc:3037] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:01.629235 10002 raft_consensus.cc:3037] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:01.629956 10002 raft_consensus.cc:2454] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91986472596b4296b00b2c72ff869011 in term 1.
I20260925 19:12:01.630072 9734 raft_consensus.cc:2454] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91986472596b4296b00b2c72ff869011 in term 1.
I20260925 19:12:01.630115 9801 leader_election.cc:302] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011; no voters:
I20260925 19:12:01.630210 10071 raft_consensus.cc:2781] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:01.630391 10071 raft_consensus.cc:686] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 1 LEADER]: Becoming Leader. State: Replica: 91986472596b4296b00b2c72ff869011, State: Running, Role: LEADER
I20260925 19:12:01.630484 10071 consensus_queue.cc:237] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.631346 9589 catalog_manager.cc:5495] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91986472596b4296b00b2c72ff869011 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "91986472596b4296b00b2c72ff869011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } health_report { overall_health: UNKNOWN } } }
W20260925 19:12:01.632853 9915 tablet.cc:2369] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:12:01.648337 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.645709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.27.174.66:41689)
0925 19:12:01.645757 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.645759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.645771 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:12:01.645847 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.645847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.645867 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.645931 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:01.646119 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.646600 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.647394 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.647643 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.647723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.647954 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:01.648063 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:12:01.648139 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.648159 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.648180 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:12:01.650375 9734 raft_consensus.cc:1261] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Refusing update from remote peer 91986472596b4296b00b2c72ff869011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:01.650741 10002 raft_consensus.cc:1261] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Refusing update from remote peer 91986472596b4296b00b2c72ff869011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:01.650786 10075 consensus_queue.cc:1035] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:01.651048 10075 consensus_queue.cc:1035] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:01.654122 10079 mvcc.cc:205] Tried to move back new op lower bound from 7333328984677048320 to 7333328984599236608. Current Snapshot: MvccSnapshot[applied={T|T < 7333328984677048320}]
I20260925 19:12:01.654417 10081 mvcc.cc:205] Tried to move back new op lower bound from 7333328984677048320 to 7333328984599236608. Current Snapshot: MvccSnapshot[applied={T|T < 7333328984677048320}]
I20260925 19:12:01.656553 9588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:01.656659 9588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:01.657414 9588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Sending DeleteTablet for 3 replicas of tablet 926b748dcedc49e8abf249db848b9f8e
I20260925 19:12:01.657780 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9651
I20260925 19:12:01.657776 9714 tablet_service.cc:1514] Processing DeleteTablet for tablet 926b748dcedc49e8abf249db848b9f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57908
I20260925 19:12:01.657779 9982 tablet_service.cc:1514] Processing DeleteTablet for tablet 926b748dcedc49e8abf249db848b9f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:42398
I20260925 19:12:01.659304 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet 926b748dcedc49e8abf249db848b9f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33026
I20260925 19:12:01.659554 10089 tablet_replica.cc:330] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: stopping tablet replica
I20260925 19:12:01.659674 10089 raft_consensus.cc:2229] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:12:01.659801 10089 raft_consensus.cc:2258] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:01.660240 10089 ts_tablet_manager.cc:1905] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:01.662320 10089 ts_tablet_manager.cc:1918] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:01.662389 10089 log.cc:1196] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/926b748dcedc49e8abf249db848b9f8e
I20260925 19:12:01.662643 10089 ts_tablet_manager.cc:1939] T 926b748dcedc49e8abf249db848b9f8e P 91986472596b4296b00b2c72ff869011: Deleting consensus metadata
I20260925 19:12:01.663224 9574 catalog_manager.cc:4841] TS 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689): tablet 926b748dcedc49e8abf249db848b9f8e (table test-table [id=cc56aff518ff40ffb1e2c30eca30e863]) successfully deleted
I20260925 19:12:01.663547 10088 tablet_replica.cc:330] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: stopping tablet replica
I20260925 19:12:01.663743 10088 raft_consensus.cc:2229] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:01.663873 10088 raft_consensus.cc:2258] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:12:01.664073 9574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:37643 (error 108)
I20260925 19:12:01.664404 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9785
I20260925 19:12:01.664590 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.664445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.0.0.1:57912)
0925 19:12:01.664515 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.664551 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 19:12:01.664692 9574 catalog_manager.cc:4584] TS 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 926b748dcedc49e8abf249db848b9f8e: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37643: connect: Connection refused (error 111)
I20260925 19:12:01.666224 10088 ts_tablet_manager.cc:1905] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:01.668013 10088 ts_tablet_manager.cc:1918] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:01.668118 10088 log.cc:1196] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/926b748dcedc49e8abf249db848b9f8e
I20260925 19:12:01.668457 10088 ts_tablet_manager.cc:1939] T 926b748dcedc49e8abf249db848b9f8e P 2f30deba566e4e1f80d3b2cf8286a993: Deleting consensus metadata
I20260925 19:12:01.668915 9573 catalog_manager.cc:4841] TS 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): tablet 926b748dcedc49e8abf249db848b9f8e (table test-table [id=cc56aff518ff40ffb1e2c30eca30e863]) successfully deleted
I20260925 19:12:01.670090 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9919
I20260925 19:12:01.675153 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 9557
I20260925 19:12:01.681048 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--webserver_interface=127.27.174.126
--webserver_port=36077
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:01.767761 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:01.768028 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:01.768069 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:01.769340 10091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:12:01.769382 10091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:01.769397 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:01.769410 10091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:12:01.769454 10091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:12:01.770900 10091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=36077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:01.771157 10091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:01.771412 10091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:01.780092 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.780102 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.780189 10091 server_base.cc:1034] running on GCE node
W20260925 19:12:01.780102 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.780675 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:01.781045 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:01.782249 10091 hybrid_clock.cc:648] HybridClock initialized: now 1790363521782227 us; error 33 us; skew 500 ppm
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:01.793393 10091 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:12:01.793998 10091 webserver.cc:460] Webserver started at http://127.27.174.126:36077/ using document root <none> and password file <none>
I20260925 19:12:01.794286 10091 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:01.794377 10091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:01.796046 10091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:12:01.796767 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.796973 10091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:12:01.797083 10091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "f77aec91b2104c15933714ce08a119dd"
format_stamp: "Formatted at 2026-09-25 19:12:00 on dist-test-slave-fcdn"
server_key: "1ed0fcf98877420d12cf7de78345e1fd"
server_key_iv: "7257689deff222f02545b0aac59e39ac"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.797538 10091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.814037 10091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.816556 10091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.816730 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.820892 10091 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43315
I20260925 19:12:01.820926 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43315 every 8 connection(s)
I20260925 19:12:01.821311 10091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:12:01.822880 10160 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:12:01.823577 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap starting.
I20260925 19:12:01.826432 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10091
I20260925 19:12:01.826754 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:37643
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=35287
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:01.832062 10160 log.cc:824] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.834054 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:01.834368 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap complete.
I20260925 19:12:01.836750 10160 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.837018 10160 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.837227 10160 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.837323 10160 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:12:01.837380 10160 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:12:01.837428 10160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:12:01.838438 10160 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.838557 10160 leader_election.cc:302] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77aec91b2104c15933714ce08a119dd; no voters:
I20260925 19:12:01.838745 10160 leader_election.cc:288] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:12:01.838820 10164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:12:01.839020 10160 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:12:01.839041 10164 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 LEADER]: Becoming Leader. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Running, Role: LEADER
I20260925 19:12:01.839274 10164 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:01.839571 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:01.839670 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:01.839838 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77aec91b2104c15933714ce08a119dd. Latest consensus state: current_term: 2 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:01.839890 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:01.840096 10173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:12:01.840937 10173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:12:01.841084 10173 catalog_manager.cc:1259] Loaded cluster ID: 7ed5827a1cc14484ba1ce97845adb8f1
I20260925 19:12:01.841360 10173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:12:01.841663 10173 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 19:12:01.847364 10173 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 19:12:01.850041 10173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:12:01.850255 10173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Loaded TSK: 0
I20260925 19:12:01.850529 10173 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:12:01.917302 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:01.917560 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:01.917603 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:01.918788 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:01.918860 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:01.918908 10162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:12:01.920495 10162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:37643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=35287
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:01.920778 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:01.921013 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:01.927464 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.927505 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:01.927549 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:01.927923 10162 server_base.cc:1034] running on GCE node
I20260925 19:12:01.928113 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:01.928411 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:01.929582 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790363521929559 us; error 37 us; skew 500 ppm
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:01.932385 10162 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:12:01.932946 10162 webserver.cc:460] Webserver started at http://127.27.174.65:35287/ using document root <none> and password file <none>
I20260925 19:12:01.933208 10162 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:01.933285 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:01.935132 10162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:12:01.936005 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:01.936232 10162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:01.936336 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "50ac90119ef5401384c49cc76a95d242"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "b38a97bf18f67bf9d6759692be073e40"
server_key_iv: "831916b22bb791e735d9e427e21c643f"
server_key_version: "encryptionkey@0"
I20260925 19:12:01.936789 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:01.952114 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:01.954968 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:01.955138 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:01.955425 10162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:01.955989 10204 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:12:01.957516 10162 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:12:01.957580 10162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:12:01.957615 10162 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:12:01.958479 10162 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:12:01.958528 10162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:12:01.958612 10204 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Bootstrap starting.
I20260925 19:12:01.965456 10162 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37643
I20260925 19:12:01.965996 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:12:01.966645 10311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37643 every 8 connection(s)
I20260925 19:12:01.969492 10204 log.cc:824] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:01.972055 10204 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:01.972365 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10162
I20260925 19:12:01.972654 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:41689
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=46491
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:01.973172 10204 tablet_bootstrap.cc:492] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Bootstrap complete.
I20260925 19:12:01.974350 10204 ts_tablet_manager.cc:1397] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260925 19:12:01.976215 10204 raft_consensus.cc:348] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
I20260925 19:12:01.976668 10204 raft_consensus.cc:729] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ac90119ef5401384c49cc76a95d242, State: Initialized, Role: FOLLOWER
I20260925 19:12:01.976831 10204 consensus_queue.cc:260] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } }
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:01.977149 10204 ts_tablet_manager.cc:1428] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 19:12:01.982138 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.966802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:36545)
0925 19:12:01.967165 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.967217 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.967987 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.970183 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.970187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.970551 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:01.970825 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.970832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.972581 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.972583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.973948 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.973952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.974759 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.974760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.974912 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.975718 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:01.975729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:01.977528 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:01.980034 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.980043 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.980045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.980503 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.980836 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.980837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.980837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.980976 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.981483 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:01.981490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:01.981649 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.981761 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.981932 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":102,"threads_started":1}
I20260925 19:12:01.982246 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.967094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:36545 (local address 127.27.174.126:43315)
0925 19:12:01.969728 (+ 2634us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.969738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.969790 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:12:01.969912 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.969918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.969966 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.970132 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.970972 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.972401 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.974059 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.974645 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.974750 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.977650 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.977654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.977654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.977667 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.979931 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.980605 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.980609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.980610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.980748 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.981054 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.981055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.981055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.981398 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.981494 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.981841 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.982007 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2567,"thread_start_us":68,"threads_started":1}
I20260925 19:12:01.983583 10312 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:01.983784 10312 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:01.983974 10312 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:01.984841 10124 ts_manager.cc:194] Registered new tserver with Master: 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643)
I20260925 19:12:01.986169 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:36545
I20260925 19:12:01.987087 10312 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
Sep 25 19:12:01 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20260925 19:12:01.994146 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.985579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.0.0.1:57916)
0925 19:12:01.985825 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:01.985861 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:01.985965 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:01.986177 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:01.986181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:01.986555 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:01.986612 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.986615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.988205 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.988206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.988777 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:01.988778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.989280 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.989281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:01.989360 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.989871 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:01.989876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:01.991139 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:01.992967 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.992973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.992975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.993173 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.993413 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:01.993414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:01.993415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:01.993479 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:01.993762 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:01.993768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:01.993820 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 19:12:01.993924 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 19:12:01.994045 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260925 19:12:01.994146 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:01.985648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.27.174.65:37643)
0925 19:12:01.985832 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:01.985837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:01.985949 (+ 112us) server_negotiation.cc:407] Connection header received
0925 19:12:01.986075 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:01.986078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:01.986097 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:01.986174 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:01.987292 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.988116 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.988878 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:01.989180 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:01.989273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:01.991238 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:01.991245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:01.991246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:01.991253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:01.992863 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.993276 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.993279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.993280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.993339 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:01.993542 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:01.993543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:01.993543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:01.993683 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:01.993783 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:01.993959 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:12:01.994049 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260925 19:12:01.994762 10246 tablet_service.cc:1514] Processing DeleteTablet for tablet 926b748dcedc49e8abf249db848b9f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57916
I20260925 19:12:01.995309 10321 tablet_replica.cc:330] stopping tablet replica
I20260925 19:12:01.995435 10321 raft_consensus.cc:2229] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:01.995527 10321 raft_consensus.cc:2258] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:01.995848 10321 ts_tablet_manager.cc:1905] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:01.997980 10321 ts_tablet_manager.cc:1918] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:01.998075 10321 log.cc:1196] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/926b748dcedc49e8abf249db848b9f8e
I20260925 19:12:01.998369 10321 ts_tablet_manager.cc:1939] T 926b748dcedc49e8abf249db848b9f8e P 50ac90119ef5401384c49cc76a95d242: Deleting consensus metadata
I20260925 19:12:01.998926 10109 catalog_manager.cc:4841] TS 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643): tablet 926b748dcedc49e8abf249db848b9f8e (table test-table [id=cc56aff518ff40ffb1e2c30eca30e863]) successfully deleted
W20260925 19:12:02.063097 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.063347 10317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.063376 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.064558 10317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.064599 10317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.064626 10317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:12:02.066025 10317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:41689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=46491
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.066242 10317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.066476 10317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.071780 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.071941 10317 server_base.cc:1034] running on GCE node
W20260925 19:12:02.071810 10326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.071787 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.072337 10317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.072607 10317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.073799 10317 hybrid_clock.cc:648] HybridClock initialized: now 1790363522073790 us; error 36 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.076550 10317 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:12:02.077028 10317 webserver.cc:460] Webserver started at http://127.27.174.66:46491/ using document root <none> and password file <none>
I20260925 19:12:02.077262 10317 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.077347 10317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.078801 10317 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:02.079548 10336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.079835 10317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:02.079914 10317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "91986472596b4296b00b2c72ff869011"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "8a835ac9cd5f6062a6ee7e488d0f203f"
server_key_iv: "223c75bac4893b350599f918b892d3a6"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.080328 10317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:02.097473 10317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.100056 10317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.100231 10317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.100495 10317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:02.100898 10317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:02.100952 10317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.100996 10317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:02.101023 10317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.106907 10317 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41689
I20260925 19:12:02.106949 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41689 every 8 connection(s)
I20260925 19:12:02.107295 10317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:12:02.111394 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10317
I20260925 19:12:02.111742 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:42783
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=35419
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:02.117626 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.108028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:42005 (local address 127.27.174.126:43315)
0925 19:12:02.108092 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.108094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.108332 (+ 238us) server_negotiation.cc:407] Connection header received
0925 19:12:02.108775 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.108776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.108788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.108852 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:02.109806 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.110302 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.111337 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.111593 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.111700 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.114465 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:02.114467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:02.114467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:02.114475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:02.116049 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.116584 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.116585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.116585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.116643 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.116962 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.116962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.116963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.117071 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:02.117123 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.117477 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.117559 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:12:02.117959 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.107985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:42005)
0925 19:12:02.108145 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.108187 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.108684 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.108928 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.108931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.109519 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:02.109710 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.109715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.110391 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.110392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.111218 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.111221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.111704 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.111705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.111865 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.112421 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:02.112431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:02.114353 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:02.116131 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:02.116139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:02.116142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:02.116485 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:02.116724 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:02.116725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:02.116725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:02.116889 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:02.117138 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:02.117144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:02.117327 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.117459 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.117657 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260925 19:12:02.118496 10450 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:02.118609 10450 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:02.118832 10450 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:02.119253 10124 ts_manager.cc:194] Registered new tserver with Master: 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
I20260925 19:12:02.119679 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:42005
W20260925 19:12:02.194842 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.195043 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.195067 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.196302 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.196348 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.196375 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:12:02.197876 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=35419
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.198114 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.198350 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.203481 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.203481 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.203727 10453 server_base.cc:1034] running on GCE node
W20260925 19:12:02.203501 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.204017 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.204280 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.205449 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790363522205447 us; error 39 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.207944 10453 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:12:02.208410 10453 webserver.cc:460] Webserver started at http://127.27.174.67:35419/ using document root <none> and password file <none>
I20260925 19:12:02.208626 10453 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.208706 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.210033 10453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:12:02.210572 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.210741 10453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.210850 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2f30deba566e4e1f80d3b2cf8286a993"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "7b539e4e9e1d3f2f14a2423ffdf757b7"
server_key_iv: "ea3211de6bfa32bb801660020c183dd2"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.211186 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:02.221061 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.222864 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.223021 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.223282 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:02.223578 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:02.223666 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.223709 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:02.223737 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.229595 10453 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42783
I20260925 19:12:02.229637 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42783 every 8 connection(s)
I20260925 19:12:02.229964 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:02.237401 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10453
I20260925 19:12:02.239589 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.230641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:49869 (local address 127.27.174.126:43315)
0925 19:12:02.230707 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.230709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.230922 (+ 213us) server_negotiation.cc:407] Connection header received
0925 19:12:02.231398 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.231399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.231410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.231472 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:02.232510 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.233000 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.233963 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.234152 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.234247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.236372 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:02.236374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:02.236374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:02.236380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:02.237992 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.238501 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.238502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.238502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.238551 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.238884 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.238885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.238885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.239004 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:02.239081 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.239405 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.239491 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:12:02.239830 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.230602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:49869)
0925 19:12:02.230765 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.230809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.231307 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.231548 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.231552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.232194 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:02.232414 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.232419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.233079 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.233080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.233877 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.233879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.234293 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.234294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.234398 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.234862 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:02.234870 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:02.236292 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:02.238070 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:02.238076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:02.238078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:02.238427 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:02.238616 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:02.238617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:02.238617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:02.238803 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:02.239084 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:02.239091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:02.239270 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.239383 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.239561 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260925 19:12:02.240370 10582 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43315
I20260925 19:12:02.240495 10582 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:02.240681 10582 heartbeater.cc:511] Master 127.27.174.126:43315 requested a full tablet report, sending...
I20260925 19:12:02.241106 10124 ts_manager.cc:194] Registered new tserver with Master: 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783)
I20260925 19:12:02.241564 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:49869
I20260925 19:12:02.244343 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.237797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.27.174.126:43315)
0925 19:12:02.237926 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.237928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.237977 (+ 49us) server_negotiation.cc:407] Connection header received
0925 19:12:02.238034 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.238034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.238047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.238105 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:02.238864 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.239427 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.240536 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.240818 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.240894 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.242230 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:02.242231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:02.242231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:02.242238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:02.243454 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.243808 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.243808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.243809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.243853 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:02.243976 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:02.243976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:02.243976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:02.244058 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:02.244166 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.244211 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.244278 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260925 19:12:02.244963 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:12:02.247306 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.245265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.27.174.126:43315)
0925 19:12:02.245296 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.245298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.245400 (+ 102us) server_negotiation.cc:407] Connection header received
0925 19:12:02.245464 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.245465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.245490 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.245544 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:02.245627 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.246056 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.246712 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.246888 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.246935 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.247056 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:02.247135 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:12:02.247189 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.247222 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.247235 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:12:02.247560 10124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:12:02.249084 10124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:12:02.251689 10246 tablet_service.cc:1467] Processing CreateTablet for tablet 796e6ffabef246ac971f77997f2310e8 (DEFAULT_TABLE table=test-table [id=08714a8295ef454a97647ba06e786c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:02.252018 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e6ffabef246ac971f77997f2310e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:02.253716 10204 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Bootstrap starting.
I20260925 19:12:02.254058 10204 tablet_bootstrap.cc:654] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:02.254604 10204 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: No bootstrap required, opened a new log
I20260925 19:12:02.254657 10204 ts_tablet_manager.cc:1397] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:12:02.255069 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.251011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42783 (local address 127.0.0.1:42412)
0925 19:12:02.251068 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.251079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.251120 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.251587 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.251588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.251590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:02.251652 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.251653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.252870 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.252871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.253842 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.253845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.254770 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.254772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.254840 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.254860 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.254949 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.254996 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:12:02.255082 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.251077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.27.174.67:42783)
0925 19:12:02.251394 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.251397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.251405 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:12:02.251461 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.251463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.251480 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.251564 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:02.251857 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.252759 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.253942 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.254664 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.254766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.254830 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.254966 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.255006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260925 19:12:02.255182 10204 raft_consensus.cc:348] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.255264 10204 raft_consensus.cc:374] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:02.255318 10204 raft_consensus.cc:729] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ac90119ef5401384c49cc76a95d242, State: Initialized, Role: FOLLOWER
I20260925 19:12:02.255378 10204 consensus_queue.cc:260] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.255486 10204 ts_tablet_manager.cc:1428] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:12:02.255962 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.251079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41689 (local address 127.0.0.1:33042)
0925 19:12:02.251237 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.251247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.251301 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.252137 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.252138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.252140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:02.252181 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.252183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.253482 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.253483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.254518 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.254521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.255535 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.255537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.255637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.255641 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.255818 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.255863 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 19:12:02.255882 10516 tablet_service.cc:1467] Processing CreateTablet for tablet 796e6ffabef246ac971f77997f2310e8 (DEFAULT_TABLE table=test-table [id=08714a8295ef454a97647ba06e786c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:02.256193 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.251505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.27.174.66:41689)
0925 19:12:02.251926 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.251930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.251948 (+ 18us) server_negotiation.cc:407] Connection header received
0925 19:12:02.251991 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.251993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.252009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.252259 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:02.252346 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.253364 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.254607 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.255369 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.255440 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.255496 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.256066 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.256112 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":103,"threads_started":1}
I20260925 19:12:02.256242 10516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e6ffabef246ac971f77997f2310e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:02.259114 10592 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Bootstrap starting.
I20260925 19:12:02.259693 10384 tablet_service.cc:1467] Processing CreateTablet for tablet 796e6ffabef246ac971f77997f2310e8 (DEFAULT_TABLE table=test-table [id=08714a8295ef454a97647ba06e786c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:02.259827 10592 tablet_bootstrap.cc:654] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:02.260015 10384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e6ffabef246ac971f77997f2310e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:02.260336 10592 log.cc:824] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:02.261188 10592 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: No bootstrap required, opened a new log
I20260925 19:12:02.261286 10592 ts_tablet_manager.cc:1397] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:12:02.262625 10592 raft_consensus.cc:348] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.262786 10592 raft_consensus.cc:374] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:02.262831 10592 raft_consensus.cc:729] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f30deba566e4e1f80d3b2cf8286a993, State: Initialized, Role: FOLLOWER
I20260925 19:12:02.262972 10592 consensus_queue.cc:260] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.263064 10594 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Bootstrap starting.
I20260925 19:12:02.263357 10592 ts_tablet_manager.cc:1428] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:02.263383 10582 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
I20260925 19:12:02.263815 10594 tablet_bootstrap.cc:654] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:02.264370 10594 log.cc:824] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:02.265101 10594 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: No bootstrap required, opened a new log
I20260925 19:12:02.265193 10594 ts_tablet_manager.cc:1397] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:02.266484 10594 raft_consensus.cc:348] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.266636 10594 raft_consensus.cc:374] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:02.266692 10594 raft_consensus.cc:729] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91986472596b4296b00b2c72ff869011, State: Initialized, Role: FOLLOWER
I20260925 19:12:02.266811 10594 consensus_queue.cc:260] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.267078 10594 ts_tablet_manager.cc:1428] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:02.267156 10450 heartbeater.cc:503] Master 127.27.174.126:43315 was elected leader, sending a full tablet report...
I20260925 19:12:02.277328 10595 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:02.277474 10595 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.277875 10595 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689), 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643)
I20260925 19:12:02.280848 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.278083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41689 (local address 127.27.174.67:43287)
0925 19:12:02.278150 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.278165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.278231 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.278416 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.278418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.278419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:02.278489 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.278497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.279140 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.279141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.279966 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.279968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.280587 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.280588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.280689 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.280694 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.280746 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.280793 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260925 19:12:02.280892 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.278177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:43287 (local address 127.27.174.66:41689)
0925 19:12:02.278224 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.278226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.278236 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:12:02.278321 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.278323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.278334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.278397 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:02.278571 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.279057 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.280056 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.280505 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.280581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.280591 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.280775 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.280822 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:12:02.281440 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.278222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.27.174.67:35289)
0925 19:12:02.278400 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.278407 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.278470 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.278636 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:02.278637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:02.278638 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:02.278713 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.278719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.279356 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.279357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.280180 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:02.280182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.281181 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.281182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:02.281238 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.281241 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:12:02.281316 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:12:02.281354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":21,"thread_start_us":110,"threads_started":1}
I20260925 19:12:02.281433 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.278301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35289 (local address 127.27.174.65:37643)
0925 19:12:02.278343 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.278344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.278460 (+ 116us) server_negotiation.cc:407] Connection header received
0925 19:12:02.278523 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.278525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.278550 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.278634 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:02.278794 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.279272 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.280292 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.280925 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.280994 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.281073 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.281323 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.281365 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:12:02.281395 10404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e6ffabef246ac971f77997f2310e8" candidate_uuid: "2f30deba566e4e1f80d3b2cf8286a993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91986472596b4296b00b2c72ff869011" is_pre_election: true
I20260925 19:12:02.281608 10404 raft_consensus.cc:2454] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f30deba566e4e1f80d3b2cf8286a993 in term 0.
I20260925 19:12:02.281708 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e6ffabef246ac971f77997f2310e8" candidate_uuid: "2f30deba566e4e1f80d3b2cf8286a993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ac90119ef5401384c49cc76a95d242" is_pre_election: true
I20260925 19:12:02.281848 10470 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011; no voters:
I20260925 19:12:02.281874 10266 raft_consensus.cc:2454] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f30deba566e4e1f80d3b2cf8286a993 in term 0.
I20260925 19:12:02.281978 10595 raft_consensus.cc:2781] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:12:02.282037 10595 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:12:02.282063 10595 raft_consensus.cc:3037] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:02.282800 10595 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.282938 10595 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [CANDIDATE]: Term 1 election: Requested vote from peers 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689), 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643)
I20260925 19:12:02.283094 10404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e6ffabef246ac971f77997f2310e8" candidate_uuid: "2f30deba566e4e1f80d3b2cf8286a993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91986472596b4296b00b2c72ff869011"
I20260925 19:12:02.283110 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e6ffabef246ac971f77997f2310e8" candidate_uuid: "2f30deba566e4e1f80d3b2cf8286a993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ac90119ef5401384c49cc76a95d242"
I20260925 19:12:02.283183 10266 raft_consensus.cc:3037] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:02.283180 10404 raft_consensus.cc:3037] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:02.284085 10404 raft_consensus.cc:2454] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f30deba566e4e1f80d3b2cf8286a993 in term 1.
I20260925 19:12:02.284085 10266 raft_consensus.cc:2454] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f30deba566e4e1f80d3b2cf8286a993 in term 1.
I20260925 19:12:02.284271 10470 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011; no voters:
I20260925 19:12:02.284399 10595 raft_consensus.cc:2781] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:02.284567 10595 raft_consensus.cc:686] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 LEADER]: Becoming Leader. State: Replica: 2f30deba566e4e1f80d3b2cf8286a993, State: Running, Role: LEADER
I20260925 19:12:02.284703 10595 consensus_queue.cc:237] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.285575 10124 catalog_manager.cc:5495] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "2f30deba566e4e1f80d3b2cf8286a993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } health_report { overall_health: UNKNOWN } } }
I20260925 19:12:02.293116 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.290758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.27.174.67:42783)
0925 19:12:02.290809 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:02.290811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:02.290822 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:12:02.290859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:02.290860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:02.290874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:02.290948 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:02.291055 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.291544 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.292324 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:02.292513 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:02.292588 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:02.292780 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:02.292871 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:12:02.292964 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:02.292984 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 19:12:02.293006 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:12:02.295086 10266 raft_consensus.cc:1261] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Refusing update from remote peer 2f30deba566e4e1f80d3b2cf8286a993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:02.295137 10404 raft_consensus.cc:1261] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Refusing update from remote peer 2f30deba566e4e1f80d3b2cf8286a993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:02.295485 10599 consensus_queue.cc:1035] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:02.295624 10595 consensus_queue.cc:1035] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:02.298151 10603 mvcc.cc:205] Tried to move back new op lower bound from 7333328987317710848 to 7333328987278974976. Current Snapshot: MvccSnapshot[applied={T|T < 7333328987317710848}]
I20260925 19:12:02.298144 10604 mvcc.cc:205] Tried to move back new op lower bound from 7333328987317710848 to 7333328987278974976. Current Snapshot: MvccSnapshot[applied={T|T < 7333328987317710848}]
I20260925 19:12:02.299943 10122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:02.300035 10122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:02.300760 10122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Sending DeleteTablet for 3 replicas of tablet 796e6ffabef246ac971f77997f2310e8
I20260925 19:12:02.301060 10516 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e6ffabef246ac971f77997f2310e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:02 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:42412
I20260925 19:12:02.301137 10384 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e6ffabef246ac971f77997f2310e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:02 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33042
I20260925 19:12:02.301169 10246 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e6ffabef246ac971f77997f2310e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:02 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57916
I20260925 19:12:02.301290 10321 tablet_replica.cc:330] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: stopping tablet replica
I20260925 19:12:02.301347 10321 raft_consensus.cc:2229] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:02.301397 10321 raft_consensus.cc:2258] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:02.301618 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10162
I20260925 19:12:02.301755 10321 ts_tablet_manager.cc:1905] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:02.303715 10611 tablet_replica.cc:330] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: stopping tablet replica
I20260925 19:12:02.303822 10611 raft_consensus.cc:2229] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:12:02.303921 10611 raft_consensus.cc:2258] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:02.304338 10612 tablet_replica.cc:330] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: stopping tablet replica
I20260925 19:12:02.304486 10612 raft_consensus.cc:2229] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:02.304570 10611 ts_tablet_manager.cc:1905] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:02.304636 10612 raft_consensus.cc:2258] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:02.305130 10612 ts_tablet_manager.cc:1905] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:02.306671 10611 ts_tablet_manager.cc:1918] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:02.306787 10612 ts_tablet_manager.cc:1918] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:02.306806 10611 log.cc:1196] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/796e6ffabef246ac971f77997f2310e8
I20260925 19:12:02.306945 10612 log.cc:1196] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/796e6ffabef246ac971f77997f2310e8
I20260925 19:12:02.307106 10611 ts_tablet_manager.cc:1939] T 796e6ffabef246ac971f77997f2310e8 P 2f30deba566e4e1f80d3b2cf8286a993: Deleting consensus metadata
I20260925 19:12:02.307313 10612 ts_tablet_manager.cc:1939] T 796e6ffabef246ac971f77997f2310e8 P 91986472596b4296b00b2c72ff869011: Deleting consensus metadata
W20260925 19:12:02.307714 10109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:37643 (error 108)
I20260925 19:12:02.307757 10108 catalog_manager.cc:4841] TS 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): tablet 796e6ffabef246ac971f77997f2310e8 (table test-table [id=08714a8295ef454a97647ba06e786c13]) successfully deleted
I20260925 19:12:02.307950 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10317
I20260925 19:12:02.308238 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.308087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:37643 (local address 127.0.0.1:57928)
0925 19:12:02.308167 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.308203 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 19:12:02.308542 10109 catalog_manager.cc:4584] TS 50ac90119ef5401384c49cc76a95d242 (127.27.174.65:37643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 796e6ffabef246ac971f77997f2310e8: Network error: Client connection negotiation failed: client connection to 127.27.174.65:37643: connect: Connection refused (error 111)
I20260925 19:12:02.313377 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10453
I20260925 19:12:02.313408 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.313301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41689 (local address 127.0.0.1:33052)
0925 19:12:02.313349 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.313371 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.66:41689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:12:02.319362 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10091
I20260925 19:12:02.324654 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--webserver_interface=127.27.174.126
--webserver_port=36077
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:02.410444 10614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.410702 10614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.410740 10614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.411936 10614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:12:02.412006 10614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.412030 10614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.412061 10614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:12:02.412109 10614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:12:02.413643 10614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=36077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.413916 10614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.414148 10614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.419097 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.419097 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.419164 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.419327 10614 server_base.cc:1034] running on GCE node
I20260925 19:12:02.419585 10614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.419919 10614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.421094 10614 hybrid_clock.cc:648] HybridClock initialized: now 1790363522421080 us; error 33 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.423919 10614 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:12:02.424369 10614 webserver.cc:460] Webserver started at http://127.27.174.126:36077/ using document root <none> and password file <none>
I20260925 19:12:02.424566 10614 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.424610 10614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.425997 10614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:02.426630 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.426815 10614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.426920 10614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "f77aec91b2104c15933714ce08a119dd"
format_stamp: "Formatted at 2026-09-25 19:12:00 on dist-test-slave-fcdn"
server_key: "1ed0fcf98877420d12cf7de78345e1fd"
server_key_iv: "7257689deff222f02545b0aac59e39ac"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.427277 10614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:02.439441 10614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.441538 10614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.441704 10614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.445618 10614 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43315
I20260925 19:12:02.445654 10681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43315 every 8 connection(s)
I20260925 19:12:02.446055 10614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:12:02.447696 10682 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:12:02.448405 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap starting.
I20260925 19:12:02.455477 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10614
I20260925 19:12:02.455829 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:37643
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=35287
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:02.456862 10682 log.cc:824] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:02.459203 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:02.459537 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd: Bootstrap complete.
I20260925 19:12:02.461897 10682 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:02.462174 10682 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Initialized, Role: FOLLOWER
I20260925 19:12:02.462380 10682 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:02.462466 10682 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:12:02.462522 10682 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:12:02.462577 10682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:12:02.463683 10682 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:02.463811 10682 leader_election.cc:302] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77aec91b2104c15933714ce08a119dd; no voters:
I20260925 19:12:02.464025 10682 leader_election.cc:288] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:12:02.464062 10686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:12:02.464171 10686 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [term 3 LEADER]: Becoming Leader. State: Replica: f77aec91b2104c15933714ce08a119dd, State: Running, Role: LEADER
I20260925 19:12:02.464309 10682 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:12:02.464327 10686 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } }
I20260925 19:12:02.464612 10686 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77aec91b2104c15933714ce08a119dd. Latest consensus state: current_term: 3 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:02.464680 10686 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:02.464828 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f77aec91b2104c15933714ce08a119dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77aec91b2104c15933714ce08a119dd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43315 } } }
I20260925 19:12:02.464900 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77aec91b2104c15933714ce08a119dd [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:02.465183 10691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:12:02.466094 10691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:12:02.466218 10691 catalog_manager.cc:1259] Loaded cluster ID: 7ed5827a1cc14484ba1ce97845adb8f1
I20260925 19:12:02.466274 10691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 19:12:02.466531 10691 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f77796ee9ad google::LogMessage::Fail() at ??:0
@ 0x7f77796f188a google::LogMessage::SendToLog() at ??:0
@ 0x7f77796ee4de google::LogMessage::Flush() at ??:0
@ 0x7f77796efa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f778160f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f77816188b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f777b5bc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f777b5b1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f777c4156db start_thread at ??:0
@ 0x7f77781ec61f clone at ??:0
W20260925 19:12:02.549017 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.549263 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.549302 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.550524 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.550581 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.550625 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:12:02.552243 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:37643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=35287
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.552524 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.552753 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.559015 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.559015 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.559099 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.559355 10684 server_base.cc:1034] running on GCE node
I20260925 19:12:02.559566 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.559870 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.561048 10684 hybrid_clock.cc:648] HybridClock initialized: now 1790363522561009 us; error 45 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.564188 10684 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:12:02.564766 10684 webserver.cc:460] Webserver started at http://127.27.174.65:35287/ using document root <none> and password file <none>
I20260925 19:12:02.565029 10684 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.565088 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.566797 10684 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:12:02.567696 10728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.567878 10684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:02.567991 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "50ac90119ef5401384c49cc76a95d242"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "b38a97bf18f67bf9d6759692be073e40"
server_key_iv: "831916b22bb791e735d9e427e21c643f"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.568357 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:02.580794 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.583144 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.583293 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.583528 10684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:02.584012 10735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:12:02.585978 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:12:02.586048 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:12:02.586093 10684 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:12:02.586699 10684 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:12:02.586773 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:12:02.586860 10735 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Bootstrap starting.
I20260925 19:12:02.589563 10735 log.cc:824] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:02.592758 10735 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:02.593084 10735 tablet_bootstrap.cc:492] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Bootstrap complete.
I20260925 19:12:02.593259 10735 ts_tablet_manager.cc:1397] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260925 19:12:02.593994 10684 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:37643
I20260925 19:12:02.594007 10843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:37643 every 8 connection(s)
I20260925 19:12:02.594365 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:12:02.595415 10735 raft_consensus.cc:348] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.595722 10735 raft_consensus.cc:729] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ac90119ef5401384c49cc76a95d242, State: Initialized, Role: FOLLOWER
I20260925 19:12:02.595887 10735 consensus_queue.cc:260] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:02.596179 10735 ts_tablet_manager.cc:1428] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:12:02.601137 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10684
I20260925 19:12:02.601438 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:41689
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=46491
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:02.720575 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.720786 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.720810 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.721973 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.722016 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.722043 10848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:12:02.723497 10848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:41689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=46491
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.723775 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.724017 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.729137 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.729220 10848 server_base.cc:1034] running on GCE node
W20260925 19:12:02.729031 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.729035 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.729496 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.729789 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.731000 10848 hybrid_clock.cc:648] HybridClock initialized: now 1790363522730970 us; error 43 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.733788 10848 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:12:02.734261 10848 webserver.cc:460] Webserver started at http://127.27.174.66:46491/ using document root <none> and password file <none>
I20260925 19:12:02.734493 10848 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.734570 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.736035 10848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:12:02.736776 10863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.736984 10848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:12:02.737095 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "91986472596b4296b00b2c72ff869011"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "8a835ac9cd5f6062a6ee7e488d0f203f"
server_key_iv: "223c75bac4893b350599f918b892d3a6"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.737468 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260925 19:12:02.743594 10846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:12:02.595017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:35507)
0925 19:12:02.595173 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.595217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:02.595395 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:02.743383 (+147988us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260925 19:12:02.743980 10844 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:43315 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:43315: Client connection negotiation failed: client connection to 127.27.174.126:43315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:12:02.744378 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.744256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:46857)
0925 19:12:02.744314 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.744352 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:12:02.744740 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.744651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:33797)
0925 19:12:02.744695 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.744717 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:12:02.744881 10844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:12:02.748903 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.751052 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.751206 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.751410 10848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:02.751801 10848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:02.751857 10848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.751902 10848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:02.751931 10848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.757954 10848 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41689
I20260925 19:12:02.758025 10976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41689 every 8 connection(s)
I20260925 19:12:02.758373 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:12:02.759469 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.759060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:36431)
0925 19:12:02.759254 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.759329 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
W20260925 19:12:02.759903 10977 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:43315 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:43315: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
I20260925 19:12:02.760363 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.760208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:42525)
0925 19:12:02.760310 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.760341 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 19:12:02.760754 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.760692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:37769)
0925 19:12:02.760716 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.760738 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 19:12:02.760890 10977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:12:02.767794 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10848
I20260925 19:12:02.768054 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:42783
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:43315
--webserver_port=35419
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:02.849355 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:02.849560 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:02.849582 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:02.850733 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:02.850773 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:02.850800 10980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:12:02.852361 10980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=35419
--tserver_master_addrs=127.27.174.126:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:02.852596 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:02.852826 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:02.857748 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.857748 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:02.857769 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:02.858188 10980 server_base.cc:1034] running on GCE node
I20260925 19:12:02.858428 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:02.858630 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:02.859762 10980 hybrid_clock.cc:648] HybridClock initialized: now 1790363522859752 us; error 27 us; skew 500 ppm
Sep 25 19:12:02 dist-test-slave-fcdn krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:02.862279 10980 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:12:02.862712 10980 webserver.cc:460] Webserver started at http://127.27.174.67:35419/ using document root <none> and password file <none>
I20260925 19:12:02.862936 10980 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:02.862982 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:02.864375 10980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:02.864953 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.865139 10980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:12:02.865227 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2f30deba566e4e1f80d3b2cf8286a993"
format_stamp: "Formatted at 2026-09-25 19:12:01 on dist-test-slave-fcdn"
server_key: "7b539e4e9e1d3f2f14a2423ffdf757b7"
server_key_iv: "ea3211de6bfa32bb801660020c183dd2"
server_key_version: "encryptionkey@0"
I20260925 19:12:02.865557 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:02.880812 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:02.882826 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:02.882966 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:02.883216 10980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:02.883524 10980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:02.883558 10980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.883581 10980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:02.883656 10980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:02.889245 10980 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42783
I20260925 19:12:02.889361 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42783 every 8 connection(s)
I20260925 19:12:02.889654 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:12:02.890677 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.890312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:38781)
0925 19:12:02.890465 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.890541 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20260925 19:12:02.891109 11109 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:43315 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:43315: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
I20260925 19:12:02.891367 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.891301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:60943)
0925 19:12:02.891328 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.891352 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 19:12:02.891638 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:02.891572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:39259)
0925 19:12:02.891593 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:02.891626 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 19:12:02.891750 11109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:12:02.892853 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 10980
I20260925 19:12:03.745554 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:03.745305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:44097)
0925 19:12:03.745448 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:03.745494 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260925 19:12:03.761516 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:03.761287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:54399)
0925 19:12:03.761422 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:03.761460 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260925 19:12:03.892364 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:03.892120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:44279)
0925 19:12:03.892275 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:03.892313 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260925 19:12:04.130714 11116 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:04.130831 11116 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:04.131139 11116 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783), 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
Sep 25 19:12:04 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.66@KRBTEST.COM
Sep 25 19:12:04 dist-test-slave-fcdn krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.67@KRBTEST.COM
I20260925 19:12:04.142366 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.131382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:55671 (local address 127.27.174.67:42783)
0925 19:12:04.131518 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:04.131529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:04.131563 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:12:04.132059 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:04.132061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:04.132116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:04.132263 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:04.133058 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.133908 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.134930 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.135331 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.135447 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:04.138263 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:04.138267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:04.138268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:04.138279 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:04.140161 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:04.140849 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:04.140853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:04.140853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:04.140986 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:04.141312 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:04.141313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:04.141313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:04.141641 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:04.141739 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:04.142109 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 19:12:04.142248 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260925 19:12:04.142376 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.131399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34657 (local address 127.27.174.66:41689)
0925 19:12:04.131549 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:04.131560 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:04.131595 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:12:04.132059 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:04.132061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:04.132116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:04.132268 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:04.132961 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.133851 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.134921 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.135322 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.135447 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:04.137974 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:04.137979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:04.137979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:04.137996 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:04.140072 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:04.140849 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:04.140853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:04.140853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:04.140986 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:04.141312 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:04.141313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:04.141313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:04.141665 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:04.141739 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:04.142109 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 19:12:04.142280 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260925 19:12:04.142385 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.131278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41689 (local address 127.27.174.65:34657)
0925 19:12:04.131460 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:04.131470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:04.131528 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:04.132304 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:04.132306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:04.132646 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:04.132843 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:04.132848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.133968 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.133969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:04.134826 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:04.134828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.135416 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.135418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:04.135555 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:04.136244 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:04.136248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:04.137842 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:04.140291 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:04.140292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:04.140292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:04.140666 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:04.141068 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:04.141069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:04.141069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:04.141233 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:04.141744 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:04.141752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:04.141918 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:12:04.142040 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:12:04.142251 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":132,"thread_start_us":67,"threads_started":1}
I20260925 19:12:04.142414 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.131271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42783 (local address 127.27.174.65:55671)
0925 19:12:04.131326 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:04.131338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:04.131393 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:04.132303 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:04.132306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:04.132875 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:04.132930 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:04.132932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.134003 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.134005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:04.134826 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:04.134828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:04.135424 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:04.135425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:04.135557 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:04.136194 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:04.136230 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:04.138137 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:04.140257 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:04.140265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:04.140268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:04.140666 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:04.141095 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:04.141096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:04.141096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:04.141242 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:04.141744 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:04.141752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:04.141918 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:12:04.142040 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:12:04.142308 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":149}
W20260925 19:12:04.143301 10730 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
W20260925 19:12:04.143576 10729 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
I20260925 19:12:04.143664 10729 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50ac90119ef5401384c49cc76a95d242; no voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011
I20260925 19:12:04.143782 11116 raft_consensus.cc:2726] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:12:04.746282 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.746037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:60917)
0925 19:12:04.746194 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:04.746234 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260925 19:12:04.762161 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.761947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:35037)
0925 19:12:04.762077 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:04.762120 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 19:12:04.893028 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:04.892808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:37383)
0925 19:12:04.892945 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:04.892985 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260925 19:12:05.705737 11123 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:05.705833 11123 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:05.706060 11123 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783), 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
W20260925 19:12:05.706563 10729 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
W20260925 19:12:05.706638 10730 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
I20260925 19:12:05.706696 10730 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50ac90119ef5401384c49cc76a95d242; no voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011
I20260925 19:12:05.706772 11123 raft_consensus.cc:2726] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:12:05.747125 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:05.746797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:55183)
0925 19:12:05.746982 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:05.747028 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260925 19:12:05.762928 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:05.762683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:56581)
0925 19:12:05.762837 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:05.762875 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260925 19:12:05.893755 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:05.893520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:50511)
0925 19:12:05.893672 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:05.893709 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260925 19:12:06.747884 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:06.747645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:59193)
0925 19:12:06.747793 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:06.747836 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260925 19:12:06.763578 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:06.763361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:50517)
0925 19:12:06.763502 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:06.763539 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260925 19:12:06.894475 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:06.894266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:51433)
0925 19:12:06.894398 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:06.894435 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260925 19:12:07.748669 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:07.748405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:55241)
0925 19:12:07.748570 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:07.748621 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260925 19:12:07.764279 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:07.764085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:35293)
0925 19:12:07.764202 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:07.764238 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 19:12:07.895164 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:07.894925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:35625)
0925 19:12:07.895062 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:07.895098 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260925 19:12:08.749390 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:08.749174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:40213)
0925 19:12:08.749299 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:08.749343 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260925 19:12:08.764951 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:08.764729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:41577)
0925 19:12:08.764871 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:08.764910 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 19:12:08.896085 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:08.895884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:43083)
0925 19:12:08.896003 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:08.896042 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 19:12:09.004918 11136 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:09.005009 11136 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:09.005211 11136 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783), 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
W20260925 19:12:09.005586 10730 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
W20260925 19:12:09.005646 10729 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
I20260925 19:12:09.005704 10730 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50ac90119ef5401384c49cc76a95d242; no voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011
I20260925 19:12:09.005792 11136 raft_consensus.cc:2726] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:12:09.750130 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:09.749915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:53999)
0925 19:12:09.750038 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:09.750080 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 19:12:09.765702 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:09.765475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:43253)
0925 19:12:09.765622 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:09.765659 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260925 19:12:09.896818 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:09.896576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:46149)
0925 19:12:09.896725 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:09.896771 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260925 19:12:10.750890 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:10.750646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:47893)
0925 19:12:10.750798 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:10.750842 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260925 19:12:10.766486 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:10.766251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:41755)
0925 19:12:10.766396 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:10.766443 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260925 19:12:10.897697 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:10.897485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:60669)
0925 19:12:10.897608 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:10.897649 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 19:12:11.751751 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:11.751463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:36443)
0925 19:12:11.751596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:11.751703 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 19:12:11.767262 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:11.767049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:48765)
0925 19:12:11.767181 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:11.767219 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260925 19:12:11.898454 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:11.898223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:42959)
0925 19:12:11.898353 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:11.898407 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260925 19:12:12.752604 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:12.752331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:34107)
0925 19:12:12.752510 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:12.752555 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260925 19:12:12.768097 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:12.767875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:42185)
0925 19:12:12.768012 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:12.768050 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 19:12:12.899314 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:12.899077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:46999)
0925 19:12:12.899228 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:12.899268 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260925 19:12:13.061167 11149 raft_consensus.cc:482] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:13.061271 11149 raft_consensus.cc:504] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f30deba566e4e1f80d3b2cf8286a993" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42783 } } peers { permanent_uuid: "91986472596b4296b00b2c72ff869011" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41689 } } peers { permanent_uuid: "50ac90119ef5401384c49cc76a95d242" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 37643 } }
I20260925 19:12:13.061497 11149 leader_election.cc:288] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783), 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689)
W20260925 19:12:13.061911 10729 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f30deba566e4e1f80d3b2cf8286a993 (127.27.174.67:42783): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
W20260925 19:12:13.061980 10730 leader_election.cc:341] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91986472596b4296b00b2c72ff869011 (127.27.174.66:41689): Not found: Tablet not found: 796e6ffabef246ac971f77997f2310e8
I20260925 19:12:13.062011 10729 leader_election.cc:302] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50ac90119ef5401384c49cc76a95d242; no voters: 2f30deba566e4e1f80d3b2cf8286a993, 91986472596b4296b00b2c72ff869011
I20260925 19:12:13.062100 11149 raft_consensus.cc:2726] T 796e6ffabef246ac971f77997f2310e8 P 50ac90119ef5401384c49cc76a95d242 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:12:13.753567 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:13.753347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:51953)
0925 19:12:13.753478 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:13.753521 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 19:12:13.768826 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:13.768574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:54103)
0925 19:12:13.768689 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:13.768781 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260925 19:12:13.900095 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:13.899845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:36141)
0925 19:12:13.900004 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:13.900049 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260925 19:12:14.754577 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:14.754276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:60339)
0925 19:12:14.754465 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:14.754518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260925 19:12:14.769629 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:14.769383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:58371)
0925 19:12:14.769527 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:14.769567 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260925 19:12:14.900959 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:14.900699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:60771)
0925 19:12:14.900836 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:14.900884 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260925 19:12:15.755646 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:15.755337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:59741)
0925 19:12:15.755503 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:15.755547 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260925 19:12:15.770434 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:15.770187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:58761)
0925 19:12:15.770353 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:15.770389 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260925 19:12:15.901845 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:15.901617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:47389)
0925 19:12:15.901759 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:15.901800 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260925 19:12:16.756484 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:16.756231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:40769)
0925 19:12:16.756396 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:16.756438 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260925 19:12:16.771307 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:16.771059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:55817)
0925 19:12:16.771212 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:16.771262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260925 19:12:16.902736 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:16.902443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:43953)
0925 19:12:16.902635 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:16.902679 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260925 19:12:17.757184 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:17.756999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.65:35713)
0925 19:12:17.757110 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:17.757141 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260925 19:12:17.771935 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:17.771755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.66:52929)
0925 19:12:17.771862 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:17.771896 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260925 19:12:17.900796 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10684
I20260925 19:12:17.903499 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:17.903299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43315 (local address 127.27.174.67:44765)
0925 19:12:17.903423 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:17.903459 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260925 19:12:17.905855 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10848
I20260925 19:12:17.909934 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 10980
2026-09-25T19:12:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18206 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:12:17 dist-test-slave-fcdn krb5kdc[11167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:12:17 dist-test-slave-fcdn krb5kdc[11167](info): set up 2 sockets
Sep 25 19:12:17 dist-test-slave-fcdn krb5kdc[11167](info): commencing operation
krb5kdc: starting...
W20260925 19:12:18.947667 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:12:19.066633 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:19.146435 11180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:19.146678 11180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:19.146724 11180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:19.147888 11180 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:12:19.147953 11180 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:12:19.147974 11180 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:12:19.147991 11180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:12:19.148025 11180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:19.148051 11180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:19.148082 11180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:12:19.148109 11180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:12:19.149667 11180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:19.149946 11180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:19.150161 11180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:19.155002 11185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.155028 11186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.155002 11188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.155025 11180 server_base.cc:1034] running on GCE node
I20260925 19:12:19.155369 11180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:19.155646 11180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:19.156819 11180 hybrid_clock.cc:648] HybridClock initialized: now 1790363539156811 us; error 30 us; skew 500 ppm
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:19.159241 11180 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:12:19.159724 11180 webserver.cc:460] Webserver started at http://127.27.174.126:39663/ using document root <none> and password file <none>
I20260925 19:12:19.159898 11180 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:19.159937 11180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:19.160022 11180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:19.160959 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "9ebcb7dc5f794018b00c1d5691f32e55"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "c60b8580759d4f128991685264897482"
server_key_iv: "d4f40e9451a7129e87545b8d3c6d4796"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.161243 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "9ebcb7dc5f794018b00c1d5691f32e55"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "c60b8580759d4f128991685264897482"
server_key_iv: "d4f40e9451a7129e87545b8d3c6d4796"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.162503 11180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:12:19.163220 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.163432 11180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:12:19.163517 11180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "9ebcb7dc5f794018b00c1d5691f32e55"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "c60b8580759d4f128991685264897482"
server_key_iv: "d4f40e9451a7129e87545b8d3c6d4796"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.163573 11180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:19.172782 11180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:19.174443 11180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:19.174546 11180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:19.178215 11180 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35189
I20260925 19:12:19.178273 11247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35189 every 8 connection(s)
I20260925 19:12:19.178586 11180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:12:19.179430 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:19.180949 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11180
I20260925 19:12:19.181051 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:12:19.181258 28345 external_mini_cluster.cc:1428] Setting key ec21afaa5fb76538a3bb42784ea35ea8
I20260925 19:12:19.182068 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap starting.
I20260925 19:12:19.182631 11248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:19.182926 11248 log.cc:824] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:19.183748 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: No bootstrap required, opened a new log
I20260925 19:12:19.185937 11248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:19.186095 11248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:19.186127 11248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Initialized, Role: FOLLOWER
I20260925 19:12:19.186245 11248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:19.186328 11248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:12:19.186360 11248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:12:19.186441 11248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:19.187071 11248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:19.187170 11248 leader_election.cc:302] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ebcb7dc5f794018b00c1d5691f32e55; no voters:
I20260925 19:12:19.187337 11248 leader_election.cc:288] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:12:19.187371 11253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:19.187481 11253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Running, Role: LEADER
I20260925 19:12:19.187623 11248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:12:19.187634 11253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:19.187994 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:12:19.188009 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ebcb7dc5f794018b00c1d5691f32e55. Latest consensus state: current_term: 1 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:12:19.188200 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:19.188226 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
W20260925 19:12:19.189574 11269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:12:19.189668 11269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:12:19.189941 11262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:12:19.190141 11262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:12:19.192113 11262 catalog_manager.cc:1347] Generated new cluster ID: 88c4a7d102d947599cd8d1cf979ed199
I20260925 19:12:19.192163 11262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:12:19.193181 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.181759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.27.174.126:35189)
0925 19:12:19.181974 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.181983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.182007 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:12:19.182477 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.182484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.182538 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.182685 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:19.183463 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.184849 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.185596 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.186201 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.186306 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.189408 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:19.189411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:19.189412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:19.189421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:19.191295 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.191741 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.191745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.191745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.191832 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.192043 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.192044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.192044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.192342 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:19.192478 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.192862 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.192984 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 19:12:19.208105 11262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:12:19.208647 11262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:12:19.211086 11262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Generated new TSK 0
I20260925 19:12:19.211262 11262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:12:19.239053 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:19.333983 11274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:19.334182 11274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:19.334205 11274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:19.335355 11274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:19.335397 11274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:19.335422 11274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:12:19.336971 11274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:19.337203 11274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:19.337442 11274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:19.342823 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.342831 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.342911 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.342918 11274 server_base.cc:1034] running on GCE node
I20260925 19:12:19.343206 11274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:19.343454 11274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:19.344615 11274 hybrid_clock.cc:648] HybridClock initialized: now 1790363539344577 us; error 52 us; skew 500 ppm
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:19.346971 11274 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:12:19.347321 11274 webserver.cc:460] Webserver started at http://127.27.174.65:44427/ using document root <none> and password file <none>
I20260925 19:12:19.347510 11274 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:19.347551 11274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:19.347748 11274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:19.348724 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "b24b6093c4c344f59955f419ddbb3c8d"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "f60e0211f5a5bd4a8f1a8e00ae0a8a31"
server_key_iv: "58d188bb6be09769704929ebb6778f48"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.349020 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "b24b6093c4c344f59955f419ddbb3c8d"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "f60e0211f5a5bd4a8f1a8e00ae0a8a31"
server_key_iv: "58d188bb6be09769704929ebb6778f48"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.350288 11274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:12:19.350943 11289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.351130 11274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:12:19.351212 11274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "b24b6093c4c344f59955f419ddbb3c8d"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "f60e0211f5a5bd4a8f1a8e00ae0a8a31"
server_key_iv: "58d188bb6be09769704929ebb6778f48"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.351281 11274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:19.378373 11274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:19.380328 11274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:19.380441 11274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:19.380667 11274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:19.381057 11274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:19.381114 11274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.381155 11274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:19.381181 11274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.387084 11274 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41507
I20260925 19:12:19.387135 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41507 every 8 connection(s)
I20260925 19:12:19.387440 11274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:19.394550 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11274
I20260925 19:12:19.394631 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:12:19.394847 28345 external_mini_cluster.cc:1428] Setting key dc24283bdf8f9760a530a42a8420a01b
I20260925 19:12:19.396876 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.388133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:40773 (local address 127.27.174.126:35189)
0925 19:12:19.388197 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.388198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.388397 (+ 199us) server_negotiation.cc:407] Connection header received
0925 19:12:19.388953 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.388954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.388964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.389015 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:19.390044 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.390541 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.391553 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.391870 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.391924 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.394017 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:19.394018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:19.394018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:19.394024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:19.395484 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.395917 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.395918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.395919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.395980 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.396207 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.396208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.396209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.396346 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:19.396413 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.396640 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.396731 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:12:19.396934 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.388128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:40773)
0925 19:12:19.388274 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.388311 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.388866 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.389076 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.389079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.389771 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:19.389962 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.389968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.390623 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.390624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.391474 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.391477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.391943 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.391944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.392047 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.392483 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:19.392490 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:19.393928 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:19.395573 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.395579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.395581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.395844 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.396050 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.396050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.396051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.396144 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.396409 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:19.396413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:19.396519 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.396598 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.396715 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260925 19:12:19.397763 11403 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:19.397857 11403 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:19.398041 11403 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:19.398574 11212 ts_manager.cc:194] Registered new tserver with Master: b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507)
I20260925 19:12:19.399509 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:40773
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:12:19.422505 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:19.502589 11408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:19.502789 11408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:19.502811 11408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:19.503895 11408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:19.503929 11408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:19.503955 11408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:12:19.505280 11408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:19.505450 11408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:19.505611 11408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:19.510596 11413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.510610 11416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.510668 11408 server_base.cc:1034] running on GCE node
W20260925 19:12:19.510608 11414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.511027 11408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:19.511276 11408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:19.512440 11408 hybrid_clock.cc:648] HybridClock initialized: now 1790363539512433 us; error 49 us; skew 500 ppm
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:19.514941 11408 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:12:19.515384 11408 webserver.cc:460] Webserver started at http://127.27.174.66:35273/ using document root <none> and password file <none>
I20260925 19:12:19.515560 11408 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:19.515652 11408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:19.515825 11408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:19.516914 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "f1888650033342a987300373daec7d7c"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "d849791db2e25c72649ed346e5de1d89"
server_key_iv: "0737024bea932fd4d4ecfb355ade09a8"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.517266 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "f1888650033342a987300373daec7d7c"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "d849791db2e25c72649ed346e5de1d89"
server_key_iv: "0737024bea932fd4d4ecfb355ade09a8"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.518559 11408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:12:19.519316 11423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.519469 11408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:12:19.519555 11408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "f1888650033342a987300373daec7d7c"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "d849791db2e25c72649ed346e5de1d89"
server_key_iv: "0737024bea932fd4d4ecfb355ade09a8"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.519651 11408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:19.548640 11408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:19.550625 11408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:19.550768 11408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:19.550966 11408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:19.551290 11408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:19.551324 11408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.551347 11408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:19.551396 11408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.557142 11408 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45689
I20260925 19:12:19.557219 11536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45689 every 8 connection(s)
I20260925 19:12:19.557554 11408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:19.567049 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.558305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38843 (local address 127.27.174.126:35189)
0925 19:12:19.558373 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.558375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.558573 (+ 198us) server_negotiation.cc:407] Connection header received
0925 19:12:19.559103 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.559104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.559116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.559157 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:19.560146 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.560660 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.561857 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.562204 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.562289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.564420 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:19.564421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:19.564422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:19.564430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:19.565673 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.566087 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.566089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.566089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.566149 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.566398 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.566399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.566399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.566517 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:19.566590 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.566878 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.566957 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:12:19.567242 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.558286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:38843)
0925 19:12:19.558459 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.558493 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.558993 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.559240 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.559244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.559854 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:19.560041 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.560046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.560775 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.560776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.561757 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.561760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.562307 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.562308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.562464 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.562909 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:19.562917 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:19.564327 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:19.565758 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.565764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.565766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.565998 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.566216 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.566217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.566218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.566330 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.566594 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:19.566600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:19.566735 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.566857 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.567016 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260925 19:12:19.567533 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11408
I20260925 19:12:19.567657 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:12:19.567767 11537 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:19.567853 11537 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:19.567886 28345 external_mini_cluster.cc:1428] Setting key f263533798c876584eb4f96ccff437a3
I20260925 19:12:19.568033 11537 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:19.568461 11211 ts_manager.cc:194] Registered new tserver with Master: f1888650033342a987300373daec7d7c (127.27.174.66:45689)
I20260925 19:12:19.568847 11211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:38843
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:12:19.592131 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:19.672188 11542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:19.672382 11542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:19.672405 11542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:19.673549 11542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:19.673590 11542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:19.673642 11542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:12:19.675046 11542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:19.675294 11542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:19.675513 11542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:19.680435 11548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:19.680431 11547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.680485 11542 server_base.cc:1034] running on GCE node
W20260925 19:12:19.680431 11550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:19.680827 11542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:19.681072 11542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:19.682233 11542 hybrid_clock.cc:648] HybridClock initialized: now 1790363539682219 us; error 28 us; skew 500 ppm
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:19.684680 11542 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:12:19.685088 11542 webserver.cc:460] Webserver started at http://127.27.174.67:36411/ using document root <none> and password file <none>
I20260925 19:12:19.685322 11542 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:19.685390 11542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:19.685536 11542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:12:19.686595 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "2440668ee6fa420fa7a92e77a2173269"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "e3172728fe86e0c227925aa22a29e9b6"
server_key_iv: "bb9663ec9606413f1e9f4026c46401bb"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.686934 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "2440668ee6fa420fa7a92e77a2173269"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "e3172728fe86e0c227925aa22a29e9b6"
server_key_iv: "bb9663ec9606413f1e9f4026c46401bb"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.688524 11542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:12:19.689410 11557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.689616 11542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:19.689710 11542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2440668ee6fa420fa7a92e77a2173269"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "e3172728fe86e0c227925aa22a29e9b6"
server_key_iv: "bb9663ec9606413f1e9f4026c46401bb"
server_key_version: "encryptionkey@0"
I20260925 19:12:19.689797 11542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:19.705196 11542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:19.707062 11542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:19.707219 11542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:19.707445 11542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:19.707788 11542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:19.707847 11542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.707890 11542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:19.707916 11542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:19.713719 11542 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46585
I20260925 19:12:19.713825 11670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46585 every 8 connection(s)
I20260925 19:12:19.714123 11542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:12:19.716784 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11542
I20260925 19:12:19.716871 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:12:19.717036 28345 external_mini_cluster.cc:1428] Setting key c93d0d02d4accae80db870880003c39c
Sep 25 19:12:19 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:19.723192 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.714843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40847 (local address 127.27.174.126:35189)
0925 19:12:19.714907 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.714910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.715132 (+ 222us) server_negotiation.cc:407] Connection header received
0925 19:12:19.715661 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.715662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.715674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.715760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:19.716658 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.717462 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.718566 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.718777 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.718833 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.720871 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:19.720872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:19.720872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:19.720878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:19.722057 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.722473 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.722474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.722474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.722526 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.722735 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.722736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.722736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.722829 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:19.722876 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.723079 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.723133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:12:19.723433 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.714829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:40847)
0925 19:12:19.715012 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.715048 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.715539 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.715826 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.715830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.716381 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:19.716570 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.716575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.717560 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.717561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.718493 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.718495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.718852 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.718852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.718946 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.719375 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:19.719383 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:19.720785 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:19.722128 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.722133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.722134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.722415 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.722587 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:19.722587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:19.722588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:19.722680 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:19.722881 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:19.722885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:19.722991 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.723070 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.723181 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260925 19:12:19.724013 11671 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:19.724119 11671 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:19.724294 11671 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:19.724633 11211 ts_manager.cc:194] Registered new tserver with Master: 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:19.725018 11211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:40847
I20260925 19:12:19.728574 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:12:19.736081 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.729907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.27.174.126:35189)
0925 19:12:19.730082 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.730084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.730184 (+ 100us) server_negotiation.cc:407] Connection header received
0925 19:12:19.730255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.730256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.730266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.730316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:19.730883 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.731364 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.732175 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.732375 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.732459 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.733600 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:19.733601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:19.733602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:19.733608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:19.734688 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.735143 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.735144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.735145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.735224 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:19.735433 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:19.735434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:19.735434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:19.735594 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:19.735736 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.735899 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.735980 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 19:12:19.737658 11211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:12:19.739215 11211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:12:19.745760 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45689 (local address 127.0.0.1:44088)
0925 19:12:19.741939 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.741949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.741990 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.742333 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.742337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.742339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:19.742412 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.742414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743791 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.743792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.744816 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.744818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745469 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.745545 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745559 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.745626 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.745680 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":81,"threads_started":1}
I20260925 19:12:19.745859 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.27.174.65:41507)
0925 19:12:19.742124 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.742129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.742139 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:12:19.742181 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.742183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.742199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.742309 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:19.742560 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743235 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.744716 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745468 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745578 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745634 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.745727 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.745776 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260925 19:12:19.746019 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.27.174.67:46585)
0925 19:12:19.742072 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.742078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.742315 (+ 237us) server_negotiation.cc:407] Connection header received
0925 19:12:19.742356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.742359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.742374 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.742449 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:19.742636 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743745 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.744841 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745490 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745549 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.745886 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.745934 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260925 19:12:19.745846 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41507 (local address 127.0.0.1:52566)
0925 19:12:19.741686 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.741725 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.741792 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.742423 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.742424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.742426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:19.742480 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.742482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743359 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.743360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.744626 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.744629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745584 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.745645 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745647 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.745698 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.745732 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260925 19:12:19.745930 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46585 (local address 127.0.0.1:46298)
0925 19:12:19.742155 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.742164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.742218 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.742516 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.742517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.742519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:19.742560 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.742562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743854 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.743854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.744750 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.744753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745636 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.745741 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745746 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.745797 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.745834 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":77,"threads_started":1}
I20260925 19:12:19.746515 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.741784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.27.174.66:45689)
0925 19:12:19.742102 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.742106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.742117 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:12:19.742162 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.742163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.742195 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.742487 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:19.742569 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.743684 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.744896 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.745382 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.745470 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.745530 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.746273 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.746433 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260925 19:12:19.747051 11605 tablet_service.cc:1467] Processing CreateTablet for tablet 77c69987daa441e1b29a195dea62dc0c (DEFAULT_TABLE table=test-table [id=00d7ca10951d4314a63d8df6c58c6422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:19.747095 11471 tablet_service.cc:1467] Processing CreateTablet for tablet 77c69987daa441e1b29a195dea62dc0c (DEFAULT_TABLE table=test-table [id=00d7ca10951d4314a63d8df6c58c6422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:19.747318 11605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c69987daa441e1b29a195dea62dc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:19.747334 11471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c69987daa441e1b29a195dea62dc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:19.748050 11337 tablet_service.cc:1467] Processing CreateTablet for tablet 77c69987daa441e1b29a195dea62dc0c (DEFAULT_TABLE table=test-table [id=00d7ca10951d4314a63d8df6c58c6422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:19.748346 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c69987daa441e1b29a195dea62dc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:19.749851 11688 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Bootstrap starting.
I20260925 19:12:19.750437 11688 tablet_bootstrap.cc:654] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:19.750730 11689 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Bootstrap starting.
I20260925 19:12:19.750988 11688 log.cc:824] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:19.751399 11689 tablet_bootstrap.cc:654] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:19.751761 11691 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap starting.
I20260925 19:12:19.751817 11688 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: No bootstrap required, opened a new log
I20260925 19:12:19.751878 11688 ts_tablet_manager.cc:1397] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:19.752009 11689 log.cc:824] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:19.752365 11691 tablet_bootstrap.cc:654] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:19.752838 11689 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: No bootstrap required, opened a new log
I20260925 19:12:19.752956 11689 ts_tablet_manager.cc:1397] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:12:19.752954 11691 log.cc:824] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:19.753705 11691 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: No bootstrap required, opened a new log
I20260925 19:12:19.753778 11691 ts_tablet_manager.cc:1397] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:19.753803 11688 raft_consensus.cc:348] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.753985 11688 raft_consensus.cc:374] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:19.754030 11688 raft_consensus.cc:729] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1888650033342a987300373daec7d7c, State: Initialized, Role: FOLLOWER
I20260925 19:12:19.754160 11688 consensus_queue.cc:260] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.754426 11688 ts_tablet_manager.cc:1428] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:12:19.754465 11537 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
I20260925 19:12:19.754922 11689 raft_consensus.cc:348] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.755095 11689 raft_consensus.cc:374] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:19.755141 11689 raft_consensus.cc:729] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2440668ee6fa420fa7a92e77a2173269, State: Initialized, Role: FOLLOWER
I20260925 19:12:19.755266 11689 consensus_queue.cc:260] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.755528 11689 ts_tablet_manager.cc:1428] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:12:19.755565 11671 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
I20260925 19:12:19.755784 11691 raft_consensus.cc:348] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.755936 11691 raft_consensus.cc:374] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:19.755959 11691 raft_consensus.cc:729] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24b6093c4c344f59955f419ddbb3c8d, State: Initialized, Role: FOLLOWER
I20260925 19:12:19.756022 11691 consensus_queue.cc:260] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.756290 11691 ts_tablet_manager.cc:1428] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:12:19.756322 11403 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
W20260925 19:12:19.808179 11538 tablet.cc:2369] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:12:19.888248 11404 tablet.cc:2369] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:12:19.964666 11672 tablet.cc:2369] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:12:19.981654 11694 raft_consensus.cc:482] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:19.981775 11694 raft_consensus.cc:504] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.982144 11694 leader_election.cc:288] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:19.985121 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.982311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41507 (local address 127.27.174.66:51343)
0925 19:12:19.982356 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.982368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.982419 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.982620 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.982621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.982623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:19.982710 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.982712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.983394 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.983396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.984279 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.984281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.984877 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.984878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.984959 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.984963 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.985017 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.985058 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":26}
I20260925 19:12:19.985143 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.982423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51343 (local address 127.27.174.65:41507)
0925 19:12:19.982475 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.982478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.982488 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:12:19.982520 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.982521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.982532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.982591 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:19.982791 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.983308 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.984364 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.984794 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.984849 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.984857 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.985029 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.985065 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:12:19.985298 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.982425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:60911 (local address 127.27.174.67:46585)
0925 19:12:19.982472 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:19.982474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:19.982602 (+ 128us) server_negotiation.cc:407] Connection header received
0925 19:12:19.982630 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:19.982631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:19.982640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:19.982702 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:19.982893 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.983426 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.984508 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.984927 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.985004 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.985013 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:19.985186 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:12:19.985228 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:12:19.985308 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:19.982341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46585 (local address 127.27.174.66:60911)
0925 19:12:19.982508 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:19.982517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:19.982580 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:19.982730 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:19.982731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:19.982732 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:19.982820 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.982822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.983528 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.983529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.984422 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:19.984424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:19.985030 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:19.985031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:19.985105 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:19.985109 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:19.985166 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:12:19.985229 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260925 19:12:19.985693 11625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c69987daa441e1b29a195dea62dc0c" candidate_uuid: "f1888650033342a987300373daec7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2440668ee6fa420fa7a92e77a2173269" is_pre_election: true
I20260925 19:12:19.985674 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c69987daa441e1b29a195dea62dc0c" candidate_uuid: "f1888650033342a987300373daec7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24b6093c4c344f59955f419ddbb3c8d" is_pre_election: true
I20260925 19:12:19.985818 11625 raft_consensus.cc:2454] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1888650033342a987300373daec7d7c in term 0.
I20260925 19:12:19.985864 11357 raft_consensus.cc:2454] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1888650033342a987300373daec7d7c in term 0.
I20260925 19:12:19.986073 11425 leader_election.cc:302] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c; no voters:
I20260925 19:12:19.986268 11694 raft_consensus.cc:2781] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:12:19.986323 11694 raft_consensus.cc:482] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:12:19.986340 11694 raft_consensus.cc:3037] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:19.987085 11694 raft_consensus.cc:504] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.987227 11694 leader_election.cc:288] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [CANDIDATE]: Term 1 election: Requested vote from peers b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:19.987388 11625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c69987daa441e1b29a195dea62dc0c" candidate_uuid: "f1888650033342a987300373daec7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2440668ee6fa420fa7a92e77a2173269"
I20260925 19:12:19.987408 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c69987daa441e1b29a195dea62dc0c" candidate_uuid: "f1888650033342a987300373daec7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24b6093c4c344f59955f419ddbb3c8d"
I20260925 19:12:19.987468 11625 raft_consensus.cc:3037] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:19.987469 11357 raft_consensus.cc:3037] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:19.988370 11357 raft_consensus.cc:2454] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1888650033342a987300373daec7d7c in term 1.
I20260925 19:12:19.988370 11625 raft_consensus.cc:2454] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1888650033342a987300373daec7d7c in term 1.
I20260925 19:12:19.988555 11425 leader_election.cc:302] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c; no voters:
I20260925 19:12:19.988713 11694 raft_consensus.cc:2781] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:19.988903 11694 raft_consensus.cc:686] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 1 LEADER]: Becoming Leader. State: Replica: f1888650033342a987300373daec7d7c, State: Running, Role: LEADER
I20260925 19:12:19.989020 11694 consensus_queue.cc:237] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:19.989915 11211 catalog_manager.cc:5495] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c reported cstate change: term changed from 0 to 1, leader changed from <none> to f1888650033342a987300373daec7d7c (127.27.174.66). New cstate: current_term: 1 leader_uuid: "f1888650033342a987300373daec7d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } health_report { overall_health: UNKNOWN } } }
I20260925 19:12:20.026657 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:20.024166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.27.174.66:45689)
0925 19:12:20.024249 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:20.024251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:20.024260 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:12:20.024294 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:20.024294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:20.024313 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:20.024380 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:20.024485 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:20.024988 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:20.025850 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:20.026042 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:20.026101 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:20.026304 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:20.026412 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:12:20.026487 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:20.026523 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 19:12:20.026543 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 19:12:20.028636 11357 raft_consensus.cc:1261] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Refusing update from remote peer f1888650033342a987300373daec7d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:20.028636 11625 raft_consensus.cc:1261] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Refusing update from remote peer f1888650033342a987300373daec7d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:20.029022 11694 consensus_queue.cc:1035] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:20.029202 11698 consensus_queue.cc:1035] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:20.031894 11701 mvcc.cc:205] Tried to move back new op lower bound from 7333329059954348032 to 7333329059795755008. Current Snapshot: MvccSnapshot[applied={T|T < 7333329059954348032}]
I20260925 19:12:20.032652 11702 mvcc.cc:205] Tried to move back new op lower bound from 7333329059954348032 to 7333329059795755008. Current Snapshot: MvccSnapshot[applied={T|T < 7333329059954348032 or (T in {7333329059954348032})}]
I20260925 19:12:20.034507 11212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:20.034601 11212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:20.035392 11212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Sending DeleteTablet for 3 replicas of tablet 77c69987daa441e1b29a195dea62dc0c
I20260925 19:12:20.035713 11471 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c69987daa441e1b29a195dea62dc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:20 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:44088
I20260925 19:12:20.035871 11337 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c69987daa441e1b29a195dea62dc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:20 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52566
I20260925 19:12:20.035939 11710 tablet_replica.cc:330] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: stopping tablet replica
I20260925 19:12:20.036029 11710 raft_consensus.cc:2229] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:12:20.036085 11605 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c69987daa441e1b29a195dea62dc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:20 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46298
I20260925 19:12:20.036154 11710 raft_consensus.cc:2258] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:20.036357 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11274
I20260925 19:12:20.036986 11710 ts_tablet_manager.cc:1905] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:20.037164 11712 tablet_replica.cc:330] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: stopping tablet replica
I20260925 19:12:20.037322 11712 raft_consensus.cc:2229] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:20.037462 11712 raft_consensus.cc:2258] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:20.037899 11712 ts_tablet_manager.cc:1905] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:20.038728 11710 ts_tablet_manager.cc:1918] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:20.038802 11710 log.cc:1196] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/77c69987daa441e1b29a195dea62dc0c
I20260925 19:12:20.039062 11710 ts_tablet_manager.cc:1939] T 77c69987daa441e1b29a195dea62dc0c P f1888650033342a987300373daec7d7c: Deleting consensus metadata
I20260925 19:12:20.039441 11197 catalog_manager.cc:4841] TS f1888650033342a987300373daec7d7c (127.27.174.66:45689): tablet 77c69987daa441e1b29a195dea62dc0c (table test-table [id=00d7ca10951d4314a63d8df6c58c6422]) successfully deleted
I20260925 19:12:20.040124 11712 ts_tablet_manager.cc:1918] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:20.040202 11712 log.cc:1196] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/77c69987daa441e1b29a195dea62dc0c
I20260925 19:12:20.040460 11712 ts_tablet_manager.cc:1939] T 77c69987daa441e1b29a195dea62dc0c P 2440668ee6fa420fa7a92e77a2173269: Deleting consensus metadata
I20260925 19:12:20.040937 11197 catalog_manager.cc:4841] TS 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): tablet 77c69987daa441e1b29a195dea62dc0c (table test-table [id=00d7ca10951d4314a63d8df6c58c6422]) successfully deleted
W20260925 19:12:20.042121 11198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:41507 (error 108)
W20260925 19:12:20.042743 11683 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:12:20.042493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41507 (local address 127.0.0.1:52572)
0925 19:12:20.042584 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:20.042596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:20.042645 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:20.042688 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:41507: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 19:12:20.042868 11198 catalog_manager.cc:4584] TS b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77c69987daa441e1b29a195dea62dc0c: Network error: Client connection negotiation failed: client connection to 127.27.174.65:41507: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 19:12:20.043006 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11408
I20260925 19:12:20.047394 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11542
I20260925 19:12:20.052320 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11180
I20260925 19:12:40.057381 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--webserver_interface=127.27.174.126
--webserver_port=39663
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:12:40.136682 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:40.136915 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:40.136940 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:40.138098 11714 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:12:40.138136 11714 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:12:40.138149 11714 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:12:40.138186 11714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:12:40.138216 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:40.138234 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:40.138260 11714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:12:40.138301 11714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:12:40.139873 11714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=39663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:40.140118 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:40.140348 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:40.148370 11720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.148377 11721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.148377 11723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:40.148689 11714 server_base.cc:1034] running on GCE node
I20260925 19:12:40.148957 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:40.149251 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:40.150416 11714 hybrid_clock.cc:648] HybridClock initialized: now 1790363560150394 us; error 37 us; skew 500 ppm
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:40.153015 11714 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:12:40.153527 11714 webserver.cc:460] Webserver started at http://127.27.174.126:39663/ using document root <none> and password file <none>
I20260925 19:12:40.153762 11714 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:40.153846 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:40.155222 11714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.155889 11730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.156060 11714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:12:40.156188 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "9ebcb7dc5f794018b00c1d5691f32e55"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "c60b8580759d4f128991685264897482"
server_key_iv: "d4f40e9451a7129e87545b8d3c6d4796"
server_key_version: "encryptionkey@0"
I20260925 19:12:40.156646 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:40.168802 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:40.170956 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:40.171116 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:40.175244 11714 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35189
I20260925 19:12:40.175277 11782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35189 every 8 connection(s)
I20260925 19:12:40.175698 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:12:40.177073 11783 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:12:40.177758 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap starting.
I20260925 19:12:40.182471 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11714
I20260925 19:12:40.182775 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:41507
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=44427
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:40.186033 11783 log.cc:824] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:40.188002 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:40.188320 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap complete.
I20260925 19:12:40.190542 11783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:40.190757 11783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Initialized, Role: FOLLOWER
I20260925 19:12:40.190925 11783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:40.190984 11783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:12:40.191048 11783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:12:40.191099 11783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:12:40.192200 11783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:40.192318 11783 leader_election.cc:302] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ebcb7dc5f794018b00c1d5691f32e55; no voters:
I20260925 19:12:40.192512 11783 leader_election.cc:288] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:12:40.192546 11787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:12:40.192706 11787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 LEADER]: Becoming Leader. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Running, Role: LEADER
I20260925 19:12:40.192776 11783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:12:40.193045 11787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:12:40.193638 11789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ebcb7dc5f794018b00c1d5691f32e55. Latest consensus state: current_term: 2 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:12:40.193732 11789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:40.193909 11788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:12:40.193910 11798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:12:40.194681 11798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:12:40.194743 11798 catalog_manager.cc:1259] Loaded cluster ID: 88c4a7d102d947599cd8d1cf979ed199
I20260925 19:12:40.194758 11798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:12:40.195478 11798 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:12:40.195590 11798 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Loaded TSK: 0
I20260925 19:12:40.203701 11788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
I20260925 19:12:40.213338 11798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Generated new TSK 1
I20260925 19:12:40.213433 11798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:12:40.281368 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:40.281570 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:40.281594 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:40.282760 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:40.282804 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:40.282829 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:12:40.284335 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:41507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=44427
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:40.284590 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:40.284842 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:40.291301 11813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.291335 11811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:40.291365 11785 server_base.cc:1034] running on GCE node
W20260925 19:12:40.291335 11810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:40.291793 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:40.292058 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:40.293215 11785 hybrid_clock.cc:648] HybridClock initialized: now 1790363560293179 us; error 32 us; skew 500 ppm
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:40.296331 11785 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:12:40.296842 11785 webserver.cc:460] Webserver started at http://127.27.174.65:44427/ using document root <none> and password file <none>
I20260925 19:12:40.297076 11785 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:40.297160 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:40.298666 11785 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.299489 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.299700 11785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.299819 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "b24b6093c4c344f59955f419ddbb3c8d"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "f60e0211f5a5bd4a8f1a8e00ae0a8a31"
server_key_iv: "58d188bb6be09769704929ebb6778f48"
server_key_version: "encryptionkey@0"
I20260925 19:12:40.300192 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:40.311774 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:40.314126 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:40.314284 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:40.314517 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:40.314953 11827 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:12:40.316944 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:12:40.317015 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:12:40.317062 11785 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:12:40.317672 11785 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:12:40.317730 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:12:40.317806 11827 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap starting.
I20260925 19:12:40.324774 11785 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41507
I20260925 19:12:40.324820 11934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41507 every 8 connection(s)
I20260925 19:12:40.325199 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:12:40.328038 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11785
I20260925 19:12:40.328346 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:45689
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=35273
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:40.328784 11827 log.cc:824] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:40.331524 11827 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:12:40.331852 11827 tablet_bootstrap.cc:492] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap complete.
I20260925 19:12:40.333099 11827 ts_tablet_manager.cc:1397] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260925 19:12:40.334517 11827 raft_consensus.cc:348] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:40.335418 11827 raft_consensus.cc:729] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24b6093c4c344f59955f419ddbb3c8d, State: Initialized, Role: FOLLOWER
I20260925 19:12:40.335660 11827 consensus_queue.cc:260] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } }
I20260925 19:12:40.336019 11827 ts_tablet_manager.cc:1428] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:40.340354 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.325950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:38131 (local address 127.27.174.126:35189)
0925 19:12:40.327764 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.327772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.327802 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:12:40.327964 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.327970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.328030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.328173 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:40.330352 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.331795 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.333454 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.334048 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.334608 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.336854 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:40.336858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:40.336858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:40.336870 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:40.338579 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.339012 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.339015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.339015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.339111 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.339383 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.339384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.339384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.339641 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:40.339727 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.340031 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.340151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":88,"threads_started":1}
I20260925 19:12:40.340354 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.325874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:38131)
0925 19:12:40.326056 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.326098 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.326530 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.328336 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.328341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.329979 (+ 1638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:40.330214 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.330220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.331952 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.331953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.333342 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.333344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.334174 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.334175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.334308 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.334954 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:40.334971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:40.336717 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:40.338683 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.338690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.338692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.338924 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.339184 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.339185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.339185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.339299 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.339715 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:40.339722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:40.339862 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.339976 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.340148 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260925 19:12:40.341174 11935 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:40.341270 11935 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:40.341440 11935 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:40.342110 11747 ts_manager.cc:194] Registered new tserver with Master: b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507)
I20260925 19:12:40.343237 11747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:38131
I20260925 19:12:40.344273 11935 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
I20260925 19:12:40.347375 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.342671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41507 (local address 127.0.0.1:48542)
0925 19:12:40.343688 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.343723 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.343819 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.344246 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.344250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.344252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:40.344357 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.344359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.345592 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.345592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.346363 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.346364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.347029 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.347030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.347119 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.347132 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.347234 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.347311 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":86,"threads_started":1}
I20260925 19:12:40.347370 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.342915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.27.174.65:41507)
0925 19:12:40.343088 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.343093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.343782 (+ 689us) server_negotiation.cc:407] Connection header received
0925 19:12:40.343918 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.343920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.344112 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.344187 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:40.344456 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.345515 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.346467 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.346940 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.347028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.347082 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.347253 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.347292 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260925 19:12:40.347951 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c69987daa441e1b29a195dea62dc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:20 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:48542
I20260925 19:12:40.348506 11944 tablet_replica.cc:330] stopping tablet replica
I20260925 19:12:40.348632 11944 raft_consensus.cc:2229] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:40.348699 11944 raft_consensus.cc:2258] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:40.349002 11944 ts_tablet_manager.cc:1905] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:40.351004 11944 ts_tablet_manager.cc:1918] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:40.351089 11944 log.cc:1196] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal/wals/77c69987daa441e1b29a195dea62dc0c
I20260925 19:12:40.351379 11944 ts_tablet_manager.cc:1939] T 77c69987daa441e1b29a195dea62dc0c P b24b6093c4c344f59955f419ddbb3c8d: Deleting consensus metadata
I20260925 19:12:40.351941 11733 catalog_manager.cc:4841] TS b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507): tablet 77c69987daa441e1b29a195dea62dc0c (table test-table [id=00d7ca10951d4314a63d8df6c58c6422]) successfully deleted
W20260925 19:12:40.421981 11940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:40.422174 11940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:40.422197 11940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:40.423348 11940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:40.423390 11940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:40.423416 11940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:12:40.424921 11940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:45689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=35273
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:40.425158 11940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:40.425403 11940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:40.430328 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.430341 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.430331 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:40.430615 11940 server_base.cc:1034] running on GCE node
I20260925 19:12:40.430817 11940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:40.431068 11940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:40.432235 11940 hybrid_clock.cc:648] HybridClock initialized: now 1790363560432227 us; error 34 us; skew 500 ppm
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:40.434604 11940 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:12:40.435066 11940 webserver.cc:460] Webserver started at http://127.27.174.66:35273/ using document root <none> and password file <none>
I20260925 19:12:40.435307 11940 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:40.435387 11940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:40.436766 11940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.437341 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.437505 11940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:12:40.437602 11940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "f1888650033342a987300373daec7d7c"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "d849791db2e25c72649ed346e5de1d89"
server_key_iv: "0737024bea932fd4d4ecfb355ade09a8"
server_key_version: "encryptionkey@0"
I20260925 19:12:40.437944 11940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:40.457226 11940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:40.459071 11940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:40.459231 11940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:40.459460 11940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:40.459811 11940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:40.459867 11940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.459906 11940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:40.459946 11940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.465991 11940 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45689
I20260925 19:12:40.466032 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45689 every 8 connection(s)
I20260925 19:12:40.466354 11940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:40.474690 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.467026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:52205 (local address 127.27.174.126:35189)
0925 19:12:40.467109 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.467111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.467282 (+ 171us) server_negotiation.cc:407] Connection header received
0925 19:12:40.467754 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.467755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.467766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.467817 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:40.468682 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.469159 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.470088 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.470297 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.470348 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.472462 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:40.472463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:40.472463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:40.472469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:40.473618 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.473963 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.473963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.473964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.474009 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.474207 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.474207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.474207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.474301 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:40.474348 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.474564 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.474629 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:12:40.474839 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.466997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:52205)
0925 19:12:40.467166 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.467199 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.467681 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.467866 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.467869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.468417 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:40.468600 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.468605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.469239 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.469240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.470013 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.470015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.470367 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.470368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.470464 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.470985 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:40.471000 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:40.472380 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:40.473688 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.473693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.473695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.473900 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.474066 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.474066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.474066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.474155 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.474353 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:40.474356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:40.474463 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.474540 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.474651 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260925 19:12:40.475184 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 11940
I20260925 19:12:40.475389 12073 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:40.475467 12073 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:40.475524 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:46585
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=36411
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:12:40.475694 12073 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:40.476120 11747 ts_manager.cc:194] Registered new tserver with Master: f1888650033342a987300373daec7d7c (127.27.174.66:45689)
I20260925 19:12:40.476635 11747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:52205
W20260925 19:12:40.556185 12076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:12:40.556384 12076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:12:40.556408 12076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:12:40.557657 12076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:12:40.557698 12076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:12:40.557751 12076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:12:40.559276 12076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:46585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=36411
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:12:40.559512 12076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:12:40.559777 12076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:12:40.564637 12081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.564689 12084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:12:40.564652 12082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:12:40.565006 12076 server_base.cc:1034] running on GCE node
I20260925 19:12:40.565245 12076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:12:40.565500 12076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:12:40.566700 12076 hybrid_clock.cc:648] HybridClock initialized: now 1790363560566685 us; error 24 us; skew 500 ppm
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:12:40.569361 12076 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:12:40.569801 12076 webserver.cc:460] Webserver started at http://127.27.174.67:36411/ using document root <none> and password file <none>
I20260925 19:12:40.570024 12076 fs_manager.cc:359] Metadata directory not provided
I20260925 19:12:40.570106 12076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:12:40.571547 12076 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.572165 12091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.572330 12076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.572431 12076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2440668ee6fa420fa7a92e77a2173269"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "e3172728fe86e0c227925aa22a29e9b6"
server_key_iv: "bb9663ec9606413f1e9f4026c46401bb"
server_key_version: "encryptionkey@0"
I20260925 19:12:40.572777 12076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:12:40.581689 12076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:12:40.583366 12076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:12:40.583518 12076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:12:40.583779 12076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:12:40.584067 12076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:12:40.584131 12076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.584172 12076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:12:40.584203 12076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.589687 12076 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46585
I20260925 19:12:40.589780 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46585 every 8 connection(s)
I20260925 19:12:40.590092 12076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:12:40.590649 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12076
Sep 25 19:12:40 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:12:40.599511 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.591115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.27.174.126:35189)
0925 19:12:40.591253 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.591255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.591875 (+ 620us) server_negotiation.cc:407] Connection header received
0925 19:12:40.591927 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.591927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.591935 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.591979 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:40.592820 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.593632 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.594751 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.595060 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.595125 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.596570 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:40.596572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:40.596572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:40.596580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:40.598378 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.598756 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.598757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.598757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.598816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.599014 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.599015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.599015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.599111 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:40.599282 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.599324 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.599397 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260925 19:12:40.600601 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.591105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52723 (local address 127.27.174.126:35189)
0925 19:12:40.591163 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.591166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.591178 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:12:40.591681 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.591682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.591691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.591743 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:40.593027 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.593747 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.595148 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.595459 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.595554 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.597596 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:40.597598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:40.597598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:40.597605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:40.599123 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.599606 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.599608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.599608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.599686 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.599962 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.599964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.599964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.600113 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:40.600170 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.600441 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.600515 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:12:40.600800 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.590851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:52723)
0925 19:12:40.591006 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.591055 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.591580 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.591862 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.591867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.592656 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:12:40.592944 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.592949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.593823 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.593824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.595027 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.595029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.595567 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.595568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.595726 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.596173 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:12:40.596180 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:12:40.597501 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:12:40.599218 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.599224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.599226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.599513 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.599761 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:12:40.599762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:12:40.599762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:12:40.599871 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:12:40.600175 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:12:40.600181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:12:40.600309 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.600420 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.600571 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260925 19:12:40.601291 12205 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35189
I20260925 19:12:40.601398 12205 heartbeater.cc:464] Registering TS with master...
I20260925 19:12:40.601552 12205 heartbeater.cc:511] Master 127.27.174.126:35189 requested a full tablet report, sending...
I20260925 19:12:40.601879 11747 ts_manager.cc:194] Registered new tserver with Master: 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:40.602187 11747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:52723
I20260925 19:12:40.611194 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:12:40.613744 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.611628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.27.174.126:35189)
0925 19:12:40.611665 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.611666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.611730 (+ 64us) server_negotiation.cc:407] Connection header received
0925 19:12:40.611794 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.611795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.611826 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.611883 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:40.611986 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.612440 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.613212 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.613394 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.613445 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.613559 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:40.613598 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 19:12:40.613702 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44818: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:12:40.613808 11938 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44818: token expired
W20260925 19:12:40.613919 11677 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.174.126:35189)
I20260925 19:12:40.618996 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.614178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.27.174.126:35189)
0925 19:12:40.614210 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.614211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.614219 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:12:40.614238 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.614239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.614249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.614299 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:12:40.614718 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.615129 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.615809 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.615971 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.616019 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.617026 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:12:40.617027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:12:40.617027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:12:40.617033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:12:40.618172 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.618475 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.618475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.618476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.618522 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:12:40.618634 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:12:40.618635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:12:40.618635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:12:40.618737 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:12:40.618805 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.618893 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.618945 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:12:40.620082 11747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:12:40.621651 11747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:12:40.623605 11854 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30cc93a5d74f27996f9605a6a37786 (DEFAULT_TABLE table=test-table [id=715af00ef5464574b63e5e9849d24973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:40.623852 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30cc93a5d74f27996f9605a6a37786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:40.625530 11827 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap starting.
I20260925 19:12:40.625942 11827 tablet_bootstrap.cc:654] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:40.626770 11827 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: No bootstrap required, opened a new log
I20260925 19:12:40.626829 11827 ts_tablet_manager.cc:1397] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:12:40.626981 11827 raft_consensus.cc:348] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.627044 11827 raft_consensus.cc:374] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:40.627091 11827 raft_consensus.cc:729] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24b6093c4c344f59955f419ddbb3c8d, State: Initialized, Role: FOLLOWER
I20260925 19:12:40.627140 11827 consensus_queue.cc:260] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.627218 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.623378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45689 (local address 127.0.0.1:36634)
0925 19:12:40.623412 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.623422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.623455 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.623969 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.623969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.623971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:40.624009 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.624010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.625052 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.625053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.626064 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.626066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.626971 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.626971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.627031 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.627035 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.627104 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.627151 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:12:40.627235 11827 ts_tablet_manager.cc:1428] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:12:40.627424 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.623555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.27.174.66:45689)
0925 19:12:40.623784 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.623789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.623799 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:12:40.623855 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.623857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.623879 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.623973 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:40.624058 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.624976 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.626139 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.626888 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.626960 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.627014 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.627117 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.627153 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260925 19:12:40.627585 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.623472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46585 (local address 127.0.0.1:49804)
0925 19:12:40.623669 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.623681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.623734 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.624058 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.624059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.624061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:40.624101 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.624103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.625373 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.625374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.626391 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.626393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.627354 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.627355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.627424 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.627428 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.627458 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.627500 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":117,"threads_started":1}
I20260925 19:12:40.627914 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.623727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.27.174.67:46585)
0925 19:12:40.623884 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.623889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.623899 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:12:40.623939 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.623941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.623955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.624169 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:40.624244 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.625269 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.626456 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.627188 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.627717 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.627778 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.627805 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.627842 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 19:12:40.628181 12007 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30cc93a5d74f27996f9605a6a37786 (DEFAULT_TABLE table=test-table [id=715af00ef5464574b63e5e9849d24973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:40.628501 12007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30cc93a5d74f27996f9605a6a37786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:40.628513 12139 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30cc93a5d74f27996f9605a6a37786 (DEFAULT_TABLE table=test-table [id=715af00ef5464574b63e5e9849d24973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:12:40.628840 12139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30cc93a5d74f27996f9605a6a37786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:12:40.631264 12215 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Bootstrap starting.
I20260925 19:12:40.631358 12216 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Bootstrap starting.
I20260925 19:12:40.631934 12215 tablet_bootstrap.cc:654] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:40.631990 12216 tablet_bootstrap.cc:654] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Neither blocks nor log segments found. Creating new log.
I20260925 19:12:40.632433 12215 log.cc:824] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:40.632508 12216 log.cc:824] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Log is configured to *not* fsync() on all Append() calls
I20260925 19:12:40.633080 12215 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: No bootstrap required, opened a new log
I20260925 19:12:40.633131 12215 ts_tablet_manager.cc:1397] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:12:40.633167 12216 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: No bootstrap required, opened a new log
I20260925 19:12:40.633229 12216 ts_tablet_manager.cc:1397] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:12:40.633745 11941 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:12:40.633850 11941 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.634128 11941 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:40.634979 12216 raft_consensus.cc:348] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.635093 12216 raft_consensus.cc:374] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:40.635114 12216 raft_consensus.cc:729] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2440668ee6fa420fa7a92e77a2173269, State: Initialized, Role: FOLLOWER
I20260925 19:12:40.635223 12216 consensus_queue.cc:260] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.635147 12215 raft_consensus.cc:348] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.635324 12215 raft_consensus.cc:374] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:12:40.635355 12215 raft_consensus.cc:729] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1888650033342a987300373daec7d7c, State: Initialized, Role: FOLLOWER
I20260925 19:12:40.635444 12215 consensus_queue.cc:260] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.635494 12205 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
I20260925 19:12:40.635722 12073 heartbeater.cc:503] Master 127.27.174.126:35189 was elected leader, sending a full tablet report...
I20260925 19:12:40.635821 12216 ts_tablet_manager.cc:1428] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:12:40.635725 12215 ts_tablet_manager.cc:1428] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:12:40.637774 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.634545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42029 (local address 127.27.174.66:45689)
0925 19:12:40.634605 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.634607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.634614 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:12:40.634634 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.634634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.634642 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.634684 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:40.634884 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.635550 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.637066 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.637433 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.637526 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.637534 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.637684 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.637716 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:12:40.637804 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.634266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45689 (local address 127.27.174.65:42029)
0925 19:12:40.634352 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.634364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.634405 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.634739 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.634740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.634741 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:40.634811 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.634813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.635832 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.635833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.636972 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.636975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.637527 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.637528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.637608 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.637615 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.637679 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.637716 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 19:12:40.638094 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30cc93a5d74f27996f9605a6a37786" candidate_uuid: "b24b6093c4c344f59955f419ddbb3c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1888650033342a987300373daec7d7c" is_pre_election: true
I20260925 19:12:40.638204 12027 raft_consensus.cc:2454] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24b6093c4c344f59955f419ddbb3c8d in term 0.
I20260925 19:12:40.638386 11822 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d, f1888650033342a987300373daec7d7c; no voters:
I20260925 19:12:40.638509 11941 raft_consensus.cc:2781] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:12:40.638561 11941 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:12:40.638608 11941 raft_consensus.cc:3037] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:40.638736 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.634426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:46371 (local address 127.27.174.67:46585)
0925 19:12:40.634463 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.634465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.635991 (+ 1526us) server_negotiation.cc:407] Connection header received
0925 19:12:40.636091 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.636092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.636100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.636128 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:12:40.636362 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.636982 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.637837 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.638395 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.638469 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.638478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.638641 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.638677 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:12:40.638762 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.634361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46585 (local address 127.27.174.65:46371)
0925 19:12:40.635928 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.635939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:12:40.636017 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:12:40.636187 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:12:40.636188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:12:40.636189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:12:40.636270 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.636272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.637053 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.637054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.637738 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:12:40.637740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.638496 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.638497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:12:40.638572 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.638576 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:12:40.638633 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:12:40.638668 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"thread_start_us":101,"threads_started":1}
I20260925 19:12:40.639039 12159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30cc93a5d74f27996f9605a6a37786" candidate_uuid: "b24b6093c4c344f59955f419ddbb3c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2440668ee6fa420fa7a92e77a2173269" is_pre_election: true
I20260925 19:12:40.639122 12159 raft_consensus.cc:2454] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24b6093c4c344f59955f419ddbb3c8d in term 0.
I20260925 19:12:40.639406 11941 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.639497 11941 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 1 election: Requested vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
I20260925 19:12:40.639691 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30cc93a5d74f27996f9605a6a37786" candidate_uuid: "b24b6093c4c344f59955f419ddbb3c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1888650033342a987300373daec7d7c"
I20260925 19:12:40.639706 12159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30cc93a5d74f27996f9605a6a37786" candidate_uuid: "b24b6093c4c344f59955f419ddbb3c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2440668ee6fa420fa7a92e77a2173269"
I20260925 19:12:40.639755 12027 raft_consensus.cc:3037] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:40.639784 12159 raft_consensus.cc:3037] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:12:40.640345 12027 raft_consensus.cc:2454] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24b6093c4c344f59955f419ddbb3c8d in term 1.
I20260925 19:12:40.640532 12159 raft_consensus.cc:2454] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24b6093c4c344f59955f419ddbb3c8d in term 1.
I20260925 19:12:40.640537 11822 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d, f1888650033342a987300373daec7d7c; no voters:
I20260925 19:12:40.640641 11941 raft_consensus.cc:2781] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:12:40.640779 11941 raft_consensus.cc:686] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 LEADER]: Becoming Leader. State: Replica: b24b6093c4c344f59955f419ddbb3c8d, State: Running, Role: LEADER
I20260925 19:12:40.640844 11941 consensus_queue.cc:237] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:12:40.641544 11747 catalog_manager.cc:5495] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65). New cstate: current_term: 1 leader_uuid: "b24b6093c4c344f59955f419ddbb3c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } health_report { overall_health: HEALTHY } } }
I20260925 19:12:40.651051 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.649070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.27.174.65:41507)
0925 19:12:40.649125 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:12:40.649127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:12:40.649134 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:12:40.649172 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:12:40.649173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:12:40.649189 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:12:40.649245 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:12:40.649340 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.649761 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.650449 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:12:40.650619 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:12:40.650665 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:12:40.650814 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:12:40.650898 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:12:40.650950 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:12:40.650987 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 19:12:40.651003 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:12:40.652491 12159 raft_consensus.cc:1261] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Refusing update from remote peer b24b6093c4c344f59955f419ddbb3c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:40.652490 12027 raft_consensus.cc:1261] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Refusing update from remote peer b24b6093c4c344f59955f419ddbb3c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:12:40.652864 11941 consensus_queue.cc:1035] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:40.653018 12222 consensus_queue.cc:1035] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:12:40.655328 12226 mvcc.cc:205] Tried to move back new op lower bound from 7333329144430399488 to 7333329144385474560. Current Snapshot: MvccSnapshot[applied={T|T < 7333329144430399488}]
I20260925 19:12:40.657253 11746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:40.657342 11746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:12:40.658116 11746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Sending DeleteTablet for 3 replicas of tablet 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:12:40.658396 12007 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f30cc93a5d74f27996f9605a6a37786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:40 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:36634
I20260925 19:12:40.658427 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11785
I20260925 19:12:40.658624 12234 tablet_replica.cc:330] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: stopping tablet replica
I20260925 19:12:40.658751 12234 raft_consensus.cc:2229] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:40.658891 12234 raft_consensus.cc:2258] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:40.659332 12234 ts_tablet_manager.cc:1905] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:40.659623 12139 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f30cc93a5d74f27996f9605a6a37786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:12:40 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:49804
I20260925 19:12:40.659830 12235 tablet_replica.cc:330] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: stopping tablet replica
I20260925 19:12:40.659986 12235 raft_consensus.cc:2229] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:12:40.660107 12235 raft_consensus.cc:2258] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:12:40.660518 12235 ts_tablet_manager.cc:1905] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:12:40.661883 12234 ts_tablet_manager.cc:1918] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:40.661972 12234 log.cc:1196] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/0f30cc93a5d74f27996f9605a6a37786
I20260925 19:12:40.662101 12235 ts_tablet_manager.cc:1918] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:12:40.662173 12235 log.cc:1196] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/0f30cc93a5d74f27996f9605a6a37786
I20260925 19:12:40.662281 12234 ts_tablet_manager.cc:1939] T 0f30cc93a5d74f27996f9605a6a37786 P f1888650033342a987300373daec7d7c: Deleting consensus metadata
I20260925 19:12:40.662467 12235 ts_tablet_manager.cc:1939] T 0f30cc93a5d74f27996f9605a6a37786 P 2440668ee6fa420fa7a92e77a2173269: Deleting consensus metadata
I20260925 19:12:40.662722 11732 catalog_manager.cc:4841] TS f1888650033342a987300373daec7d7c (127.27.174.66:45689): tablet 0f30cc93a5d74f27996f9605a6a37786 (table test-table [id=715af00ef5464574b63e5e9849d24973]) successfully deleted
I20260925 19:12:40.662832 11732 catalog_manager.cc:4841] TS 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): tablet 0f30cc93a5d74f27996f9605a6a37786 (table test-table [id=715af00ef5464574b63e5e9849d24973]) successfully deleted
W20260925 19:12:40.664734 11733 connection.cc:535] client connection to 127.27.174.65:41507 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 19:12:40.664842 11733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:12:40.665122 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11940
I20260925 19:12:40.665294 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:12:40.665139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:41507 (local address 127.0.0.1:48552)
0925 19:12:40.665204 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:12:40.665248 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 19:12:40.665922 11733 catalog_manager.cc:4584] TS b24b6093c4c344f59955f419ddbb3c8d (127.27.174.65:41507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f30cc93a5d74f27996f9605a6a37786: Network error: Client connection negotiation failed: client connection to 127.27.174.65:41507: connect: Connection refused (error 111)
I20260925 19:12:40.669982 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12076
I20260925 19:12:40.674413 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 11714
I20260925 19:13:00.679894 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--webserver_interface=127.27.174.126
--webserver_port=39663
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:00.762118 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:00.762343 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:00.762367 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:00.763505 12237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:13:00.763550 12237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:13:00.763563 12237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:13:00.763576 12237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:00.763588 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:00.763674 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:00.763696 12237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:00.763726 12237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:00.765211 12237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=39663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:00.765409 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:00.765594 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:00.771085 12237 server_base.cc:1034] running on GCE node
W20260925 19:13:00.771044 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:00.771097 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:00.771049 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:00.771680 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:00.772013 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:00.773201 12237 hybrid_clock.cc:648] HybridClock initialized: now 1790363580773184 us; error 48 us; skew 500 ppm
Sep 25 19:13:00 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:00.775880 12237 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:00.776383 12237 webserver.cc:460] Webserver started at http://127.27.174.126:39663/ using document root <none> and password file <none>
I20260925 19:13:00.776609 12237 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:00.776678 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:00.778321 12237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:00.779163 12252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:00.779346 12237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:00.779445 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "9ebcb7dc5f794018b00c1d5691f32e55"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "c60b8580759d4f128991685264897482"
server_key_iv: "d4f40e9451a7129e87545b8d3c6d4796"
server_key_version: "encryptionkey@0"
I20260925 19:13:00.779886 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:00.793756 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:00.796198 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:00.796376 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:00.800621 12237 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35189
I20260925 19:13:00.800664 12304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35189 every 8 connection(s)
I20260925 19:13:00.801072 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:00.802608 12305 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:13:00.803287 12305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap starting.
I20260925 19:13:00.805332 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12237
I20260925 19:13:00.805641 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:41507
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=44427
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:00.811851 12305 log.cc:824] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:00.814191 12305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:13:00.814525 12305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Bootstrap complete.
I20260925 19:13:00.816944 12305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:13:00.817198 12305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Initialized, Role: FOLLOWER
I20260925 19:13:00.817384 12305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:13:00.817462 12305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:00.817513 12305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:00.817572 12305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:13:00.818589 12305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:13:00.818710 12305 leader_election.cc:302] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ebcb7dc5f794018b00c1d5691f32e55; no voters:
I20260925 19:13:00.818933 12305 leader_election.cc:288] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:13:00.818967 12309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:13:00.819090 12309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [term 3 LEADER]: Becoming Leader. State: Replica: 9ebcb7dc5f794018b00c1d5691f32e55, State: Running, Role: LEADER
I20260925 19:13:00.819209 12305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:00.819243 12309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } }
I20260925 19:13:00.819639 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:13:00.819729 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:00.819952 12311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ebcb7dc5f794018b00c1d5691f32e55. Latest consensus state: current_term: 3 leader_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebcb7dc5f794018b00c1d5691f32e55" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35189 } } }
I20260925 19:13:00.820080 12311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:00.820207 12316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:00.821303 12316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:00.821465 12316 catalog_manager.cc:1259] Loaded cluster ID: 88c4a7d102d947599cd8d1cf979ed199
I20260925 19:13:00.821698 12316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:00.822494 12316 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:00.822695 12316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Loaded TSK: 0
I20260925 19:13:00.822753 12316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ebcb7dc5f794018b00c1d5691f32e55: Loaded TSK: 1
F20260925 19:13:00.822981 12316 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fda60af19ad google::LogMessage::Fail() at ??:0
@ 0x7fda60af488a google::LogMessage::SendToLog() at ??:0
@ 0x7fda60af14de google::LogMessage::Flush() at ??:0
@ 0x7fda60af2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda6325e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fda6325bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fda68a4b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fda68a4b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fda68a4b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fda68a11c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fda68a1bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda629bf0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda629b4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda638186db start_thread at ??:0
@ 0x7fda5f5ef61f clone at ??:0
W20260925 19:13:00.909245 12307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:00.909451 12307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:00.909475 12307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:00.910676 12307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:00.910727 12307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:00.910754 12307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:00.912292 12307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:41507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=44427
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:00.912534 12307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:00.912782 12307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:00.917850 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:00.917968 12307 server_base.cc:1034] running on GCE node
W20260925 19:13:00.917835 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:00.918128 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:00.918392 12307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:00.918669 12307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:00.919826 12307 hybrid_clock.cc:648] HybridClock initialized: now 1790363580919815 us; error 27 us; skew 500 ppm
Sep 25 19:13:00 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:00.922328 12307 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:13:00.922801 12307 webserver.cc:460] Webserver started at http://127.27.174.65:44427/ using document root <none> and password file <none>
I20260925 19:13:00.923030 12307 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:00.923100 12307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:00.924573 12307 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:00.925325 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:00.925496 12307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:00.925598 12307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "b24b6093c4c344f59955f419ddbb3c8d"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "f60e0211f5a5bd4a8f1a8e00ae0a8a31"
server_key_iv: "58d188bb6be09769704929ebb6778f48"
server_key_version: "encryptionkey@0"
I20260925 19:13:00.925971 12307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:00.945523 12307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:00.948072 12307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:00.948252 12307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:00.948503 12307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:00.948971 12363 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:13:00.951021 12307 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:13:00.951113 12307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:13:00.951182 12307 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:13:00.951932 12307 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:13:00.951988 12307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:00.952064 12363 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap starting.
I20260925 19:13:00.959695 12307 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:41507
I20260925 19:13:00.959761 12470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:41507 every 8 connection(s)
I20260925 19:13:00.960211 12307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:00.960971 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12307
I20260925 19:13:00.961266 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:45689
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=35273
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:00.962970 12363 log.cc:824] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:00.965214 12363 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:13:00.965513 12363 tablet_bootstrap.cc:492] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Bootstrap complete.
I20260925 19:13:00.966485 12363 ts_tablet_manager.cc:1397] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260925 19:13:00.967504 12363 raft_consensus.cc:348] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:00.967835 12363 raft_consensus.cc:729] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24b6093c4c344f59955f419ddbb3c8d, State: Initialized, Role: FOLLOWER
I20260925 19:13:00.967993 12363 consensus_queue.cc:260] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:00.968284 12363 ts_tablet_manager.cc:1428] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260925 19:13:01.036231 12473 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:00.960940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:35561)
0925 19:13:00.963247 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:00.963295 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:00.963951 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:01.036006 (+ 72055us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2260,"thread_start_us":85,"threads_started":1}
W20260925 19:13:01.036466 12471 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:35189 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:35189: Client connection negotiation failed: client connection to 127.27.174.126:35189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:13:01.036834 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.036715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:53319)
0925 19:13:01.036765 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.036806 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:13:01.037204 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.037115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:53573)
0925 19:13:01.037166 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.037189 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:13:01.037333 12471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:13:01.059345 12474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:01.059566 12474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:01.059592 12474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:01.060916 12474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:01.060964 12474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:01.060992 12474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:01.062532 12474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:45689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=35273
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:01.062798 12474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:01.063041 12474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:01.068446 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:01.068497 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:01.068461 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:01.068898 12474 server_base.cc:1034] running on GCE node
I20260925 19:13:01.069118 12474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:01.069357 12474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:01.070565 12474 hybrid_clock.cc:648] HybridClock initialized: now 1790363581070528 us; error 37 us; skew 500 ppm
Sep 25 19:13:01 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363581, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:01.073625 12474 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:13:01.074193 12474 webserver.cc:460] Webserver started at http://127.27.174.66:35273/ using document root <none> and password file <none>
I20260925 19:13:01.074456 12474 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:01.074535 12474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:01.076083 12474 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:01.076782 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.077001 12474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:01.077126 12474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "f1888650033342a987300373daec7d7c"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "d849791db2e25c72649ed346e5de1d89"
server_key_iv: "0737024bea932fd4d4ecfb355ade09a8"
server_key_version: "encryptionkey@0"
I20260925 19:13:01.077538 12474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:01.093844 12474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:01.096285 12474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:01.096451 12474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:01.096704 12474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:01.097018 12474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:01.097072 12474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.097124 12474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:01.097155 12474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.103327 12474 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45689
I20260925 19:13:01.103374 12604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45689 every 8 connection(s)
I20260925 19:13:01.103792 12474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:13:01.104840 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.104451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:60735)
0925 19:13:01.104607 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.104685 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260925 19:13:01.105319 12605 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:35189 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:35189: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
I20260925 19:13:01.105773 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.105668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:46781)
0925 19:13:01.105721 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.105754 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:13:01.106088 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.106012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:40445)
0925 19:13:01.106050 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.106072 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 19:13:01.106240 12605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:13:01.107874 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12474
I20260925 19:13:01.108094 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:46585
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:35189
--webserver_port=36411
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:01.192766 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:01.193013 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:01.193050 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:01.194442 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:01.194545 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:01.194597 12608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:01.196240 12608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:46585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=36411
--tserver_master_addrs=127.27.174.126:35189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:01.196506 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:01.196779 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:01.201997 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:01.202073 12608 server_base.cc:1034] running on GCE node
W20260925 19:13:01.202023 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:01.202023 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:01.202466 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:01.202726 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:01.203930 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790363581203915 us; error 30 us; skew 500 ppm
Sep 25 19:13:01 dist-test-slave-fcdn krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363581, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:01.206670 12608 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:13:01.207166 12608 webserver.cc:460] Webserver started at http://127.27.174.67:36411/ using document root <none> and password file <none>
I20260925 19:13:01.207408 12608 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:01.207458 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:01.208918 12608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:01.209542 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.209735 12608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:01.209831 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "2440668ee6fa420fa7a92e77a2173269"
format_stamp: "Formatted at 2026-09-25 19:12:19 on dist-test-slave-fcdn"
server_key: "e3172728fe86e0c227925aa22a29e9b6"
server_key_iv: "bb9663ec9606413f1e9f4026c46401bb"
server_key_version: "encryptionkey@0"
I20260925 19:13:01.210192 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:01.220336 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:01.222242 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:01.222419 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:01.222672 12608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:01.222987 12608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:01.223022 12608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.223070 12608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:01.223102 12608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:01.229597 12608 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46585
I20260925 19:13:01.229671 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46585 every 8 connection(s)
I20260925 19:13:01.230041 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:01.231108 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.230717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:45911)
0925 19:13:01.230867 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.230970 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260925 19:13:01.231519 12737 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:35189 (0 consecutive failures): Network error: Failed to ping master at 127.27.174.126:35189: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
I20260925 19:13:01.231832 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.231741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:44669)
0925 19:13:01.231788 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.231815 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:13:01.232146 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:01.232070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:58525)
0925 19:13:01.232110 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:01.232131 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 19:13:01.232261 12737 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:13:01.232862 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12608
I20260925 19:13:02.038030 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.037740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:34243)
0925 19:13:02.037880 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:02.037950 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 19:13:02.106863 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.106637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:34245)
0925 19:13:02.106761 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:02.106801 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260925 19:13:02.108783 12743 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:02.108889 12743 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:02.109295 12743 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
Sep 25 19:13:02 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.66@KRBTEST.COM
Sep 25 19:13:02 dist-test-slave-fcdn krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.67@KRBTEST.COM
I20260925 19:13:02.120997 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.109664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46585 (local address 127.27.174.65:57103)
0925 19:13:02.109756 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:02.109766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:02.109821 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:02.110585 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:02.110586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:02.110969 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:02.111147 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:02.111153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.112499 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.112501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:02.113773 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:02.113776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.114471 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.114472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:02.114569 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:02.115249 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:02.115261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:02.117421 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:02.119543 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:02.119544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:02.119545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:02.119813 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:02.120078 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:02.120079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:02.120079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:02.120137 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:02.120552 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:02.120552 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:02.120591 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 19:13:02.120683 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:13:02.120888 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":35,"thread_start_us":53,"threads_started":1}
I20260925 19:13:02.121050 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.109571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34679 (local address 127.27.174.66:45689)
0925 19:13:02.109802 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:02.109813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:02.109847 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:13:02.110316 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:02.110318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:02.110366 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:02.110537 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:02.111278 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.112309 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.113902 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.114341 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.114455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:02.117042 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:02.117046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:02.117046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:02.117057 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:02.119085 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:02.119576 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:02.119580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:02.119580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:02.119734 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:02.120034 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:02.120035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:02.120035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:02.120368 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:02.120429 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:02.120815 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 19:13:02.120943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":61,"threads_started":1}
I20260925 19:13:02.121069 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.109780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57103 (local address 127.27.174.67:46585)
0925 19:13:02.109948 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:02.109964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:02.109981 (+ 17us) server_negotiation.cc:407] Connection header received
0925 19:13:02.110384 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:02.110386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:02.110433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:02.110604 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:02.111270 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.112366 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.113859 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.114367 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.114497 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:02.117599 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:02.117602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:02.117603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:02.117613 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:02.119435 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:02.119894 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:02.119897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:02.119898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:02.120010 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:02.120221 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:02.120222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:02.120222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:02.120479 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:02.120558 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:02.120796 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 19:13:02.120952 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 19:13:02.121099 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.109471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45689 (local address 127.27.174.65:34679)
0925 19:13:02.109549 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:02.109565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:02.109641 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:02.110577 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:02.110580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:02.110969 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:02.111147 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:02.111153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.112561 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.112561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:02.113773 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:02.113776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:02.114439 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:02.114441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:02.114568 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:02.115258 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:02.115263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:02.116907 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:02.119174 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:02.119182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:02.119185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:02.119494 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:02.119809 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:02.119809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:02.119809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:02.119963 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:02.120437 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:02.120444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:02.120591 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 19:13:02.120708 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 19:13:02.120970 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":157}
W20260925 19:13:02.122135 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:02.122208 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:02.122226 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:02.122323 12743 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:02.232817 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:02.232608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:57507)
0925 19:13:02.232745 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:02.232783 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260925 19:13:03.038810 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:03.038581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:59887)
0925 19:13:03.038723 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:03.038764 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260925 19:13:03.107519 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:03.107280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:41563)
0925 19:13:03.107441 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:03.107477 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260925 19:13:03.233603 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:03.233367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:48403)
0925 19:13:03.233513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:03.233546 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260925 19:13:03.868579 12751 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:03.868703 12751 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:03.868871 12751 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
W20260925 19:13:03.869275 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:03.869352 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:03.869369 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:03.869437 12751 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:04.039759 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:04.039424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:49397)
0925 19:13:04.039591 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:04.039710 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260925 19:13:04.108361 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:04.108110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:54273)
0925 19:13:04.108274 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:04.108316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260925 19:13:04.234289 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:04.234095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:60883)
0925 19:13:04.234210 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:04.234246 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 19:13:05.040496 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:05.040274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:38283)
0925 19:13:05.040413 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:05.040448 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260925 19:13:05.109213 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:05.108959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:34151)
0925 19:13:05.109122 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:05.109165 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260925 19:13:05.234979 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:05.234761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:44711)
0925 19:13:05.234896 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:05.234935 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260925 19:13:05.659761 12758 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:05.659890 12758 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:05.660087 12758 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
W20260925 19:13:05.660517 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:05.660601 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:05.660619 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:05.660727 12758 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:06.041254 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:06.041036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:37805)
0925 19:13:06.041169 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:06.041205 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260925 19:13:06.110014 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:06.109797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:36787)
0925 19:13:06.109924 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:06.109968 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 19:13:06.235787 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:06.235479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:37645)
0925 19:13:06.235688 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:06.235738 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260925 19:13:07.042006 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:07.041776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:51679)
0925 19:13:07.041907 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:07.041955 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260925 19:13:07.110808 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:07.110568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:47431)
0925 19:13:07.110717 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:07.110761 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260925 19:13:07.236469 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:07.236272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:39685)
0925 19:13:07.236391 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:07.236427 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 19:13:08.042701 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:08.042493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:54395)
0925 19:13:08.042617 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:08.042656 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260925 19:13:08.111509 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:08.111312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:32845)
0925 19:13:08.111430 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:08.111464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 19:13:08.237198 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:08.236966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:57209)
0925 19:13:08.237100 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:08.237141 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260925 19:13:09.043473 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:09.043219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:35179)
0925 19:13:09.043380 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:09.043425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260925 19:13:09.112201 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:09.111981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:56737)
0925 19:13:09.112117 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:09.112154 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260925 19:13:09.237792 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:09.237591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:58759)
0925 19:13:09.237711 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:09.237747 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260925 19:13:10.044150 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:10.043937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:39343)
0925 19:13:10.044071 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:10.044106 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
I20260925 19:13:10.112989 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:10.112715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:45485)
0925 19:13:10.112890 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:10.112943 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260925 19:13:10.238615 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:10.238394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:39181)
0925 19:13:10.238533 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:10.238571 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260925 19:13:10.504628 12774 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:10.504729 12774 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:10.504904 12774 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
W20260925 19:13:10.505277 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:10.505371 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:10.505388 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:10.505460 12774 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:11.044843 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:11.044627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:44075)
0925 19:13:11.044762 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:11.044797 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260925 19:13:11.113777 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:11.113559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:35161)
0925 19:13:11.113676 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:11.113717 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260925 19:13:11.239337 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:11.239097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:39909)
0925 19:13:11.239253 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:11.239293 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 19:13:12.045492 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:12.045261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:60881)
0925 19:13:12.045410 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:12.045446 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260925 19:13:12.114516 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:12.114260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:55123)
0925 19:13:12.114420 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:12.114463 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260925 19:13:12.240129 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:12.239888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:58141)
0925 19:13:12.240035 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:12.240083 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 19:13:12.703840 12781 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:12.703930 12781 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:12.704129 12781 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
W20260925 19:13:12.704561 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:12.704646 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:12.704682 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:12.704777 12781 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:13.046175 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:13.045952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:53891)
0925 19:13:13.046093 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:13.046129 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260925 19:13:13.115226 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:13.115017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:42275)
0925 19:13:13.115141 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:13.115180 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260925 19:13:13.240752 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:13.240560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:42619)
0925 19:13:13.240676 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:13.240709 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260925 19:13:14.046921 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:14.046685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:55979)
0925 19:13:14.046836 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:14.046876 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260925 19:13:14.115888 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:14.115693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:55527)
0925 19:13:14.115803 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:14.115835 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260925 19:13:14.241389 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:14.241169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:36467)
0925 19:13:14.241314 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:14.241348 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260925 19:13:14.842041 12788 raft_consensus.cc:482] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:14.842132 12788 raft_consensus.cc:504] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1888650033342a987300373daec7d7c" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45689 } } peers { permanent_uuid: "2440668ee6fa420fa7a92e77a2173269" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46585 } } peers { permanent_uuid: "b24b6093c4c344f59955f419ddbb3c8d" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 41507 } }
I20260925 19:13:14.842317 12788 leader_election.cc:288] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1888650033342a987300373daec7d7c (127.27.174.66:45689), 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585)
W20260925 19:13:14.842847 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2440668ee6fa420fa7a92e77a2173269 (127.27.174.67:46585): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
W20260925 19:13:14.842944 12358 leader_election.cc:341] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1888650033342a987300373daec7d7c (127.27.174.66:45689): Not found: Tablet not found: 0f30cc93a5d74f27996f9605a6a37786
I20260925 19:13:14.842983 12358 leader_election.cc:302] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b24b6093c4c344f59955f419ddbb3c8d; no voters: 2440668ee6fa420fa7a92e77a2173269, f1888650033342a987300373daec7d7c
I20260925 19:13:14.843093 12788 raft_consensus.cc:2726] T 0f30cc93a5d74f27996f9605a6a37786 P b24b6093c4c344f59955f419ddbb3c8d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:13:15.047587 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:15.047382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:44025)
0925 19:13:15.047507 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:15.047543 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260925 19:13:15.116561 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:15.116334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:45111)
0925 19:13:15.116468 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:15.116515 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260925 19:13:15.242066 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:15.241857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:54089)
0925 19:13:15.241981 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:15.242023 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260925 19:13:16.048329 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:16.048096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.65:57907)
0925 19:13:16.048239 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:16.048273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260925 19:13:16.117265 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:16.117058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.66:53507)
0925 19:13:16.117180 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:16.117218 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260925 19:13:16.242743 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:16.242542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35189 (local address 127.27.174.67:53793)
0925 19:13:16.242666 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:16.242701 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.126:35189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260925 19:13:16.243325 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12307
I20260925 19:13:16.248104 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12474
I20260925 19:13:16.252246 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12608
2026-09-25T19:13:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58342 ms)
[----------] 30 tests from SecurityITest (229703 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:16 dist-test-slave-fcdn krb5kdc[12797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:16 dist-test-slave-fcdn krb5kdc[12797](info): set up 2 sockets
Sep 25 19:13:16 dist-test-slave-fcdn krb5kdc[12797](info): commencing operation
krb5kdc: starting...
W20260925 19:13:17.283176 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:17.405861 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43375
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:17.487941 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:17.488186 12810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:17.488222 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:17.489364 12810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:17.489431 12810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:17.489454 12810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:17.489473 12810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:17.489500 12810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:17.490974 12810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:17.491245 12810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:17.491461 12810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:17.496541 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.496528 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.496532 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:17.496573 12810 server_base.cc:1034] running on GCE node
I20260925 19:13:17.497087 12810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:17.497393 12810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:17.498553 12810 hybrid_clock.cc:648] HybridClock initialized: now 1790363597498538 us; error 27 us; skew 500 ppm
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:17.501070 12810 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:17.501529 12810 webserver.cc:460] Webserver started at http://127.27.174.126:46045/ using document root <none> and password file <none>
I20260925 19:13:17.501772 12810 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:17.501837 12810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:17.501976 12810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:17.503026 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "9c4bc7aaa1e546b69dc3d74c3225350d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "0d55205de9794622cae992fea0704cd2"
server_key_iv: "51ece8e64bdd1060b71c27c9b4a0658f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.503381 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "9c4bc7aaa1e546b69dc3d74c3225350d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "0d55205de9794622cae992fea0704cd2"
server_key_iv: "51ece8e64bdd1060b71c27c9b4a0658f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.504740 12810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:17.505507 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.505698 12810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:17.505785 12810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "9c4bc7aaa1e546b69dc3d74c3225350d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "0d55205de9794622cae992fea0704cd2"
server_key_iv: "51ece8e64bdd1060b71c27c9b4a0658f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.505865 12810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:17.533489 12810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:17.535889 12810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:17.536052 12810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:17.539915 12810 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43375
I20260925 19:13:17.539951 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43375 every 8 connection(s)
I20260925 19:13:17.540323 12810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:17.540544 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12810
I20260925 19:13:17.540656 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:17.540819 28345 external_mini_cluster.cc:1428] Setting key 277f0a77c3536c08e0c3b8d48a5a66f8
I20260925 19:13:17.541257 12878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:17.543869 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: Bootstrap starting.
I20260925 19:13:17.544437 12878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:17.544740 12878 log.cc:824] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:17.545377 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: No bootstrap required, opened a new log
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:17.547887 12878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } }
I20260925 19:13:17.548061 12878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:17.548099 12878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4bc7aaa1e546b69dc3d74c3225350d, State: Initialized, Role: FOLLOWER
I20260925 19:13:17.548234 12878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } }
I20260925 19:13:17.548328 12878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:17.548365 12878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:17.548414 12878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:17.549116 12878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } }
I20260925 19:13:17.549227 12878 leader_election.cc:302] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c4bc7aaa1e546b69dc3d74c3225350d; no voters:
I20260925 19:13:17.549394 12878 leader_election.cc:288] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:17.549460 12883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:17.549647 12878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:17.549659 12883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [term 1 LEADER]: Becoming Leader. State: Replica: 9c4bc7aaa1e546b69dc3d74c3225350d, State: Running, Role: LEADER
I20260925 19:13:17.551033 12883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } }
I20260925 19:13:17.551293 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:17.541301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.27.174.126:43375)
0925 19:13:17.541469 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:17.541478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:17.541513 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:13:17.541943 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:17.541950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:17.541995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:17.542137 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:17.542756 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.543706 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.544544 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.544997 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.545103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:17.547645 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:17.547649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:17.547649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:17.547660 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:17.549688 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.550089 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.550092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.550092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.550211 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.550424 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.550426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.550426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.550740 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:17.550919 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:17.550988 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 19:13:17.551124 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260925 19:13:17.551631 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } } }
I20260925 19:13:17.552109 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [sys.catalog]: This master's current role is: LEADER
W20260925 19:13:17.552114 12897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:17.552196 12897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:17.551765 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c4bc7aaa1e546b69dc3d74c3225350d. Latest consensus state: current_term: 1 leader_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4bc7aaa1e546b69dc3d74c3225350d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43375 } } }
I20260925 19:13:17.552461 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:17.552500 12899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:17.552670 12899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:17.553952 12899 catalog_manager.cc:1347] Generated new cluster ID: a165e7ec83aa4423907a2da1d486af40
I20260925 19:13:17.554004 12899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:17.582257 12899 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:17.582983 12899 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:17.585479 12899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: Generated new TSK 0
I20260925 19:13:17.585659 12899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:13:17.622509 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:17.724058 12904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:17.724253 12904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:17.724275 12904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:17.725354 12904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:17.725387 12904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:17.725412 12904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:17.726795 12904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:17.726979 12904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:17.727159 12904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:17.732189 12909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.732183 12912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.732185 12910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:17.732491 12904 server_base.cc:1034] running on GCE node
I20260925 19:13:17.732707 12904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:17.732903 12904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:17.734042 12904 hybrid_clock.cc:648] HybridClock initialized: now 1790363597734032 us; error 27 us; skew 500 ppm
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:17.736670 12904 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:13:17.737116 12904 webserver.cc:460] Webserver started at http://127.27.174.65:43821/ using document root <none> and password file <none>
I20260925 19:13:17.737375 12904 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:17.737440 12904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:17.737576 12904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:17.738613 12904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "f96b4d6ed1094ed78c9b3d799d0395cb"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "c9a7dbe8fec040ac35bcf4919d088837"
server_key_iv: "8fe27d44315c58d3057ca484d45dfc7f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.738967 12904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "f96b4d6ed1094ed78c9b3d799d0395cb"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "c9a7dbe8fec040ac35bcf4919d088837"
server_key_iv: "8fe27d44315c58d3057ca484d45dfc7f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.740370 12904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:17.741178 12919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.741387 12904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:17.741487 12904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "f96b4d6ed1094ed78c9b3d799d0395cb"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "c9a7dbe8fec040ac35bcf4919d088837"
server_key_iv: "8fe27d44315c58d3057ca484d45dfc7f"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.741559 12904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:17.758356 12904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:17.760706 12904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:17.760872 12904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:17.761103 12904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:17.761408 12904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:17.761458 12904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.761497 12904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:17.761528 12904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.766952 12904 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44947
I20260925 19:13:17.767026 13032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44947 every 8 connection(s)
I20260925 19:13:17.767356 12904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:17.767866 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 12904
I20260925 19:13:17.767983 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:17.768175 28345 external_mini_cluster.cc:1428] Setting key e38df1c2d4ea6a861f96debbb722a21d
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:13:17.778087 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:17.768108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34061 (local address 127.27.174.126:43375)
0925 19:13:17.768162 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:17.768164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:17.768417 (+ 253us) server_negotiation.cc:407] Connection header received
0925 19:13:17.768904 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:17.768905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:17.768915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:17.768959 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:17.770272 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.771074 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.772587 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.772850 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.772907 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:17.775255 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:17.775257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:17.775257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:17.775281 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:17.776701 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.777089 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.777090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.777090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.777149 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.777386 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.777387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.777388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.777570 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:17.777637 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:17.777914 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 19:13:17.777998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:17.778267 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:17.768097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43375 (local address 127.27.174.65:34061)
0925 19:13:17.768266 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:17.768316 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:17.768802 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:17.769035 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:17.769040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:17.769867 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:17.770168 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:17.770175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.771175 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.771176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:17.772508 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:17.772511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.772938 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.772940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:17.773093 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:17.773591 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:17.773602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:17.775164 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:17.776792 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:17.776798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:17.776800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:17.777017 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:17.777214 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:17.777215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:17.777216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:17.777323 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:17.777629 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:17.777636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:17.777782 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 19:13:17.777881 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:13:17.778039 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260925 19:13:17.778801 13033 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43375
I20260925 19:13:17.778916 13033 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:17.779135 13033 heartbeater.cc:511] Master 127.27.174.126:43375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20260925 19:13:17.779770 12842 ts_manager.cc:194] Registered new tserver with Master: f96b4d6ed1094ed78c9b3d799d0395cb (127.27.174.65:44947)
I20260925 19:13:17.780380 12842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:34061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:13:17.795154 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:17.875763 13038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:17.875952 13038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:17.875978 13038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:17.877121 13038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:17.877161 13038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:17.877210 13038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:17.878620 13038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:17.878854 13038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:17.879076 13038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:17.883926 13046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.883911 13044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:17.883900 13043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:17.884141 13038 server_base.cc:1034] running on GCE node
I20260925 19:13:17.884368 13038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:17.884593 13038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:17.885725 13038 hybrid_clock.cc:648] HybridClock initialized: now 1790363597885718 us; error 30 us; skew 500 ppm
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:17.888206 13038 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:13:17.888622 13038 webserver.cc:460] Webserver started at http://127.27.174.66:43311/ using document root <none> and password file <none>
I20260925 19:13:17.888847 13038 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:17.888911 13038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:17.889050 13038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:17.890079 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "7f29421a096948aab0f24ba99e31a11d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "f8314f5ab3be0af75c85c9cc7d62d14c"
server_key_iv: "805cfa078dfef010d85b5909c54a85ae"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.890425 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "7f29421a096948aab0f24ba99e31a11d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "f8314f5ab3be0af75c85c9cc7d62d14c"
server_key_iv: "805cfa078dfef010d85b5909c54a85ae"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.891741 13038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:13:17.892414 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.892577 13038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:17.892668 13038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "7f29421a096948aab0f24ba99e31a11d"
format_stamp: "Formatted at 2026-09-25 19:13:17 on dist-test-slave-fcdn"
server_key: "f8314f5ab3be0af75c85c9cc7d62d14c"
server_key_iv: "805cfa078dfef010d85b5909c54a85ae"
server_key_version: "encryptionkey@0"
I20260925 19:13:17.892743 13038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:17.903342 13038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:17.905469 13038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:17.905614 13038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:17.905817 13038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:17.906087 13038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:17.906136 13038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.906173 13038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:17.906203 13038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:17.911849 13038 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:35095
I20260925 19:13:17.911921 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:35095 every 8 connection(s)
I20260925 19:13:17.912215 13038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:13:17 dist-test-slave-fcdn krb5kdc[12797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363597, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:17.920037 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13038
I20260925 19:13:17.920148 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:17.920385 28345 external_mini_cluster.cc:1428] Setting key d21b6570999420dd76afe3e65748fb66
I20260925 19:13:17.920972 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:17.912897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38857 (local address 127.27.174.126:43375)
0925 19:13:17.912962 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:17.912964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:17.913174 (+ 210us) server_negotiation.cc:407] Connection header received
0925 19:13:17.913670 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:17.913671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:17.913682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:17.913743 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:17.914597 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.915074 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.916176 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.916367 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.916432 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:17.918336 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:17.918337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:17.918337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:17.918345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:17.919507 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.919919 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.919921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.919921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.919979 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:17.920277 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:17.920279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:17.920279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:17.920435 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:17.920514 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:17.920799 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 19:13:17.920884 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:13:17.921181 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:17.912882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43375 (local address 127.27.174.66:38857)
0925 19:13:17.913045 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:17.913085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:17.913586 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:17.913798 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:17.913801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:17.914341 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:17.914524 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:17.914528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.915152 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.915153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:17.916038 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:17.916041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:17.916442 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:17.916443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:17.916542 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:17.916967 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:17.916974 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:17.918241 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:17.919588 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:17.919595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:17.919597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:17.919837 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:17.920045 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:17.920046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:17.920047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:17.920172 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:17.920521 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:17.920527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:17.920644 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 19:13:17.920761 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 19:13:17.920919 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260925 19:13:17.921607 13167 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43375
I20260925 19:13:17.921712 13167 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:17.921873 13167 heartbeater.cc:511] Master 127.27.174.126:43375 requested a full tablet report, sending...
I20260925 19:13:17.922231 12842 ts_manager.cc:194] Registered new tserver with Master: 7f29421a096948aab0f24ba99e31a11d (127.27.174.66:35095)
I20260925 19:13:17.922726 12842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:38857
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:13:17.946364 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:18.026007 13172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:18.026216 13172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:18.026242 13172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:18.027400 13172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:18.027442 13172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:18.027469 13172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:18.028914 13172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:18.029158 13172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:18.029376 13172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:18.034339 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.034433 13172 server_base.cc:1034] running on GCE node
W20260925 19:13:18.034446 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.034499 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.034729 13172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:18.034970 13172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:18.036131 13172 hybrid_clock.cc:648] HybridClock initialized: now 1790363598036106 us; error 36 us; skew 500 ppm
Sep 25 19:13:18 dist-test-slave-fcdn krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:18.038473 13172 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:13:18.038889 13172 webserver.cc:460] Webserver started at http://127.27.174.67:46857/ using document root <none> and password file <none>
I20260925 19:13:18.039113 13172 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:18.039176 13172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:18.039319 13172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:18.040410 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "65cbfbb9654b4eaa89ac056225876d38"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "46cb348d1b0ecb1c0e009295a2585dfd"
server_key_iv: "8997814dc465165c8c24d86726b2a249"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.040769 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "65cbfbb9654b4eaa89ac056225876d38"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "46cb348d1b0ecb1c0e009295a2585dfd"
server_key_iv: "8997814dc465165c8c24d86726b2a249"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.042049 13172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:13:18.042747 13187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.042919 13172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:18.043015 13172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "65cbfbb9654b4eaa89ac056225876d38"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "46cb348d1b0ecb1c0e009295a2585dfd"
server_key_iv: "8997814dc465165c8c24d86726b2a249"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.043082 13172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:18.069414 13172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:18.071697 13172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:18.071852 13172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:18.072095 13172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:18.072413 13172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:18.072468 13172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.072504 13172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:18.072535 13172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.078099 13172 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42695
I20260925 19:13:18.078135 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42695 every 8 connection(s)
I20260925 19:13:18.078449 13172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:18.081058 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13172
I20260925 19:13:18.081147 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:18.081311 28345 external_mini_cluster.cc:1428] Setting key 6ce11ea73124e136242ab8bf887277d7
Sep 25 19:13:18 dist-test-slave-fcdn krb5kdc[12797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:18.087759 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.079155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35015 (local address 127.27.174.126:43375)
0925 19:13:18.079232 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.079235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.079410 (+ 175us) server_negotiation.cc:407] Connection header received
0925 19:13:18.079978 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.079979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.079991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.080043 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.080980 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.081634 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.082764 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.082963 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.083021 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.085005 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.085006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.085006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:18.085012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.086515 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:18.086873 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:18.086874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:18.086874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:18.086927 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:18.087158 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:18.087159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:18.087159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:18.087276 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.087342 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.087564 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.087693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:13:18.087900 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.079115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43375 (local address 127.27.174.67:35015)
0925 19:13:18.079280 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.079315 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.079875 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.080115 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.080118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.080676 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:18.080866 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.080872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.081778 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.081779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.082693 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.082695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.083038 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.083039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.083139 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.083591 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:18.083629 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:18.084922 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:18.086594 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:18.086600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:18.086601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:18.086811 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:18.086983 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:18.086983 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:18.086983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:18.087079 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:18.087339 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:18.087343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:18.087446 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.087523 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.087695 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260925 19:13:18.088287 13301 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43375
I20260925 19:13:18.088362 13301 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:18.088543 13301 heartbeater.cc:511] Master 127.27.174.126:43375 requested a full tablet report, sending...
I20260925 19:13:18.088955 12842 ts_manager.cc:194] Registered new tserver with Master: 65cbfbb9654b4eaa89ac056225876d38 (127.27.174.67:42695)
I20260925 19:13:18.089342 12842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:35015
I20260925 19:13:18.093173 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:18.100459 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.094507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.174.126:43375)
0925 19:13:18.094576 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.094578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.094665 (+ 87us) server_negotiation.cc:407] Connection header received
0925 19:13:18.094708 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.094708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.094717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.094771 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.095367 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.095951 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.096774 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.096953 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.097014 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.098135 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.098136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.098137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:18.098143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.099432 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:18.099784 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:18.099785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:18.099785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:18.099835 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:18.100028 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:18.100028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:18.100029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:18.100185 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.100260 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.100346 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.100400 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:13:18.101922 12842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:18.103426 12842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:18.109782 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:35095 (local address 127.0.0.1:60534)
0925 19:13:18.105985 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.105994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.106031 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.106279 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.106282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.106284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.106349 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.106351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107712 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.107713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.108709 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.108712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109572 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.109632 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109635 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.109684 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.109712 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260925 19:13:18.109829 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.27.174.66:35095)
0925 19:13:18.106025 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.106029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.106042 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:13:18.106100 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.106102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.106116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.106195 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.106433 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107575 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.108808 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109490 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109574 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109632 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.109697 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.109733 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260925 19:13:18.109783 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42695 (local address 127.0.0.1:46858)
0925 19:13:18.105777 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.105815 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.105880 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.106279 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.106282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.106287 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.106349 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.106351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107708 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.107709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.108777 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.108779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109456 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.109538 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109554 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.109628 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.109689 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":36,"thread_start_us":54,"threads_started":1}
I20260925 19:13:18.110146 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44947 (local address 127.0.0.1:35254)
0925 19:13:18.106090 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.106098 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.106162 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.106410 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.106410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.106412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.106457 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.106458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107316 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.107317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.108602 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.108604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109694 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.109762 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109767 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.109816 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.110081 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":64,"threads_started":1}
I20260925 19:13:18.110157 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.27.174.67:42695)
0925 19:13:18.106040 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.106043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.106053 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:13:18.106092 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.106094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.106110 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.106195 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.106433 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107539 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.108852 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109360 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109443 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109504 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.109822 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.110098 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":82,"threads_started":1}
I20260925 19:13:18.110442 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.105845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.27.174.65:44947)
0925 19:13:18.106028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.106033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.106166 (+ 133us) server_negotiation.cc:407] Connection header received
0925 19:13:18.106237 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.106239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.106253 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.106325 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.106535 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.107217 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.108684 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.109545 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.109613 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.109659 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.110319 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.110362 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260925 19:13:18.110819 13101 tablet_service.cc:1467] Processing CreateTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a (DEFAULT_TABLE table=test-table [id=e3688b627fe24c72a3331acb30b198e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.110942 13235 tablet_service.cc:1467] Processing CreateTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a (DEFAULT_TABLE table=test-table [id=e3688b627fe24c72a3331acb30b198e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.111214 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.111274 13235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.111224 12967 tablet_service.cc:1467] Processing CreateTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a (DEFAULT_TABLE table=test-table [id=e3688b627fe24c72a3331acb30b198e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.111513 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.113899 13318 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Bootstrap starting.
I20260925 19:13:18.114554 13318 tablet_bootstrap.cc:654] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.114590 13320 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Bootstrap starting.
I20260925 19:13:18.115275 13320 tablet_bootstrap.cc:654] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.115275 13318 log.cc:824] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.115774 13319 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Bootstrap starting.
I20260925 19:13:18.115933 13320 log.cc:824] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.116077 13318 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: No bootstrap required, opened a new log
I20260925 19:13:18.116161 13318 ts_tablet_manager.cc:1397] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:18.116370 13319 tablet_bootstrap.cc:654] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.116858 13320 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: No bootstrap required, opened a new log
I20260925 19:13:18.116928 13320 ts_tablet_manager.cc:1397] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:18.116997 13319 log.cc:824] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.117901 13319 tablet_bootstrap.cc:492] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: No bootstrap required, opened a new log
I20260925 19:13:18.117970 13319 ts_tablet_manager.cc:1397] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:18.118000 13318 raft_consensus.cc:348] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.118178 13318 raft_consensus.cc:374] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.118219 13318 raft_consensus.cc:729] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f29421a096948aab0f24ba99e31a11d, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.118333 13318 consensus_queue.cc:260] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.118592 13318 ts_tablet_manager.cc:1428] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:13:18.118732 13167 heartbeater.cc:503] Master 127.27.174.126:43375 was elected leader, sending a full tablet report...
I20260925 19:13:18.118820 13320 raft_consensus.cc:348] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.118981 13320 raft_consensus.cc:374] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.119016 13320 raft_consensus.cc:729] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96b4d6ed1094ed78c9b3d799d0395cb, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.119131 13320 consensus_queue.cc:260] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.119382 13320 ts_tablet_manager.cc:1428] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:13:18.119407 13033 heartbeater.cc:503] Master 127.27.174.126:43375 was elected leader, sending a full tablet report...
I20260925 19:13:18.120023 13319 raft_consensus.cc:348] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.120200 13319 raft_consensus.cc:374] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.120239 13319 raft_consensus.cc:729] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cbfbb9654b4eaa89ac056225876d38, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.120350 13319 consensus_queue.cc:260] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.120553 13319 ts_tablet_manager.cc:1428] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:18.120620 13301 heartbeater.cc:503] Master 127.27.174.126:43375 was elected leader, sending a full tablet report...
W20260925 19:13:18.162966 13168 tablet.cc:2369] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:13:18.268788 13034 tablet.cc:2369] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:18.302804 13325 raft_consensus.cc:482] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:18.302970 13325 raft_consensus.cc:504] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.303463 13325 leader_election.cc:288] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f29421a096948aab0f24ba99e31a11d (127.27.174.66:35095), 65cbfbb9654b4eaa89ac056225876d38 (127.27.174.67:42695)
I20260925 19:13:18.306664 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.303646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42695 (local address 127.27.174.65:59027)
0925 19:13:18.303784 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.303794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.303877 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.304069 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.304070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.304072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.304147 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.304152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.305027 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.305028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.305791 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.305793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.306383 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.306384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.306455 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.306458 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.306528 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.306569 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":55,"threads_started":1}
I20260925 19:13:18.306660 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.303716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:59027 (local address 127.27.174.67:42695)
0925 19:13:18.303764 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.303766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.303847 (+ 81us) server_negotiation.cc:407] Connection header received
0925 19:13:18.303941 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.303942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.303951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.304005 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.304228 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.304955 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.305885 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.306282 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.306382 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.306391 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.306543 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.306585 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:13:18.307252 13255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f114e75ed6ee4e2ab9c2fe2e6caaeb6a" candidate_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cbfbb9654b4eaa89ac056225876d38" is_pre_election: true
I20260925 19:13:18.307415 13255 raft_consensus.cc:2454] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96b4d6ed1094ed78c9b3d799d0395cb in term 0.
I20260925 19:13:18.307668 12920 leader_election.cc:302] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65cbfbb9654b4eaa89ac056225876d38, f96b4d6ed1094ed78c9b3d799d0395cb; no voters:
I20260925 19:13:18.307791 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.303705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:46613 (local address 127.27.174.66:35095)
0925 19:13:18.303754 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.303756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.303871 (+ 115us) server_negotiation.cc:407] Connection header received
0925 19:13:18.303994 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.303995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.304005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.304059 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.304277 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.305088 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.306657 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.307312 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.307363 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.307372 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.307639 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.307704 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:18.307818 13325 raft_consensus.cc:2781] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:18.307862 13325 raft_consensus.cc:482] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:18.307891 13325 raft_consensus.cc:3037] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:18.308080 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.303668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:35095 (local address 127.27.174.65:46613)
0925 19:13:18.303803 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.303814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.303886 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.304064 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.304065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.304068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.304174 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.304177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.305196 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.305197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.306527 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.306531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.307440 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.307441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.307531 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.307534 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.307922 (+ 388us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.307973 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260925 19:13:18.308418 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f114e75ed6ee4e2ab9c2fe2e6caaeb6a" candidate_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f29421a096948aab0f24ba99e31a11d" is_pre_election: true
I20260925 19:13:18.308575 13121 raft_consensus.cc:2454] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96b4d6ed1094ed78c9b3d799d0395cb in term 0.
I20260925 19:13:18.308900 13325 raft_consensus.cc:504] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.309046 13325 leader_election.cc:288] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [CANDIDATE]: Term 1 election: Requested vote from peers 7f29421a096948aab0f24ba99e31a11d (127.27.174.66:35095), 65cbfbb9654b4eaa89ac056225876d38 (127.27.174.67:42695)
I20260925 19:13:18.309157 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f114e75ed6ee4e2ab9c2fe2e6caaeb6a" candidate_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f29421a096948aab0f24ba99e31a11d"
I20260925 19:13:18.309239 13121 raft_consensus.cc:3037] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:18.309268 13255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f114e75ed6ee4e2ab9c2fe2e6caaeb6a" candidate_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cbfbb9654b4eaa89ac056225876d38"
I20260925 19:13:18.309336 13255 raft_consensus.cc:3037] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:18.310171 13255 raft_consensus.cc:2454] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96b4d6ed1094ed78c9b3d799d0395cb in term 1.
I20260925 19:13:18.310171 13121 raft_consensus.cc:2454] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96b4d6ed1094ed78c9b3d799d0395cb in term 1.
I20260925 19:13:18.310326 12921 leader_election.cc:302] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f29421a096948aab0f24ba99e31a11d, f96b4d6ed1094ed78c9b3d799d0395cb; no voters:
I20260925 19:13:18.310420 13325 raft_consensus.cc:2781] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:18.310562 13325 raft_consensus.cc:686] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [term 1 LEADER]: Becoming Leader. State: Replica: f96b4d6ed1094ed78c9b3d799d0395cb, State: Running, Role: LEADER
I20260925 19:13:18.310664 13325 consensus_queue.cc:237] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } }
I20260925 19:13:18.311457 12841 catalog_manager.cc:5495] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb reported cstate change: term changed from 0 to 1, leader changed from <none> to f96b4d6ed1094ed78c9b3d799d0395cb (127.27.174.65). New cstate: current_term: 1 leader_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96b4d6ed1094ed78c9b3d799d0395cb" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44947 } health_report { overall_health: HEALTHY } } }
W20260925 19:13:18.329025 13302 tablet.cc:2369] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:18.334960 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.332389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.27.174.65:44947)
0925 19:13:18.332451 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.332453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.332465 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:13:18.332547 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.332548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.332569 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.332663 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:18.332755 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.333457 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.334180 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.334470 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.334539 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.334645 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:18.334776 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:18.334854 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.334871 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.334891 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:13:18.337028 13255 raft_consensus.cc:1261] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 1 FOLLOWER]: Refusing update from remote peer f96b4d6ed1094ed78c9b3d799d0395cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:18.337042 13121 raft_consensus.cc:1261] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 1 FOLLOWER]: Refusing update from remote peer f96b4d6ed1094ed78c9b3d799d0395cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:18.337358 13329 consensus_queue.cc:1035] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f29421a096948aab0f24ba99e31a11d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:18.337505 13325 consensus_queue.cc:1035] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cbfbb9654b4eaa89ac056225876d38" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:18.339699 13332 mvcc.cc:205] Tried to move back new op lower bound from 7333329298785742848 to 7333329298681143296. Current Snapshot: MvccSnapshot[applied={T|T < 7333329298785742848}]
I20260925 19:13:18.340054 13333 mvcc.cc:205] Tried to move back new op lower bound from 7333329298785742848 to 7333329298681143296. Current Snapshot: MvccSnapshot[applied={T|T < 7333329298785742848}]
I20260925 19:13:18.340255 13334 mvcc.cc:205] Tried to move back new op lower bound from 7333329298785742848 to 7333329298681143296. Current Snapshot: MvccSnapshot[applied={T|T < 7333329298785742848}]
I20260925 19:13:18.341826 12842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:18.341941 12842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:18.342705 12842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4bc7aaa1e546b69dc3d74c3225350d: Sending DeleteTablet for 3 replicas of tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a
I20260925 19:13:18.343029 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35254
I20260925 19:13:18.343021 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60534
I20260925 19:13:18.343099 13235 tablet_service.cc:1514] Processing DeleteTablet for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:46858
I20260925 19:13:18.343390 13343 tablet_replica.cc:330] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: stopping tablet replica
I20260925 19:13:18.343393 13344 tablet_replica.cc:330] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: stopping tablet replica
I20260925 19:13:18.343477 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12904
I20260925 19:13:18.343490 13344 raft_consensus.cc:2229] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:18.343515 13342 tablet_replica.cc:330] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P f96b4d6ed1094ed78c9b3d799d0395cb: stopping tablet replica
I20260925 19:13:18.343582 13344 raft_consensus.cc:2258] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:18.345005 13343 raft_consensus.cc:2229] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:18.345129 13343 raft_consensus.cc:2258] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:18.345227 13344 ts_tablet_manager.cc:1905] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:18.345532 13343 ts_tablet_manager.cc:1905] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:18.347163 13344 ts_tablet_manager.cc:1918] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:18.347263 13344 log.cc:1196] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/f114e75ed6ee4e2ab9c2fe2e6caaeb6a
I20260925 19:13:18.347436 13343 ts_tablet_manager.cc:1918] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:18.347510 13343 log.cc:1196] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/f114e75ed6ee4e2ab9c2fe2e6caaeb6a
I20260925 19:13:18.347580 13344 ts_tablet_manager.cc:1939] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 65cbfbb9654b4eaa89ac056225876d38: Deleting consensus metadata
I20260925 19:13:18.347843 13343 ts_tablet_manager.cc:1939] T f114e75ed6ee4e2ab9c2fe2e6caaeb6a P 7f29421a096948aab0f24ba99e31a11d: Deleting consensus metadata
I20260925 19:13:18.348031 12826 catalog_manager.cc:4841] TS 65cbfbb9654b4eaa89ac056225876d38 (127.27.174.67:42695): tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a (table test-table [id=e3688b627fe24c72a3331acb30b198e4]) successfully deleted
I20260925 19:13:18.348227 12827 catalog_manager.cc:4841] TS 7f29421a096948aab0f24ba99e31a11d (127.27.174.66:35095): tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a (table test-table [id=e3688b627fe24c72a3331acb30b198e4]) successfully deleted
W20260925 19:13:18.349575 12829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:44947 (error 108)
I20260925 19:13:18.350155 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.349967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44947 (local address 127.0.0.1:35266)
0925 19:13:18.350026 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.350069 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:13:18.350183 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13038
W20260925 19:13:18.350275 12829 catalog_manager.cc:4584] TS f96b4d6ed1094ed78c9b3d799d0395cb (127.27.174.65:44947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f114e75ed6ee4e2ab9c2fe2e6caaeb6a: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44947: connect: Connection refused (error 111)
I20260925 19:13:18.355175 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13172
I20260925 19:13:18.360045 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 12810
2026-09-25T19:13:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T19:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:18Z Disabled control of system clock
I20260925 19:13:18.387102 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36683
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:18.466099 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:18.466321 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:18.466341 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:18.467423 13349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:18.467455 13349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:18.467468 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:18.467481 13349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:18.467494 13349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:18.468755 13349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:36683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:18.468937 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:18.469156 13349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:18.473945 13349 server_base.cc:1034] running on GCE node
W20260925 19:13:18.473934 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.474031 13354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.473938 13355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.474421 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:18.474706 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:18.475869 13349 hybrid_clock.cc:648] HybridClock initialized: now 1790363598475855 us; error 30 us; skew 500 ppm
I20260925 19:13:18.477083 13349 webserver.cc:460] Webserver started at http://127.27.174.126:43489/ using document root <none> and password file <none>
I20260925 19:13:18.477326 13349 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:18.477394 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:18.477555 13349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:18.478601 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "616aba1e234b4a65a171c9c0fdff8efd"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "26c291450a002f86f9e7581245b0998b"
server_key_iv: "69eb1ab171d22c792ee5d99208a9df67"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.478956 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "616aba1e234b4a65a171c9c0fdff8efd"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "26c291450a002f86f9e7581245b0998b"
server_key_iv: "69eb1ab171d22c792ee5d99208a9df67"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.480365 13349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:18.481170 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.481326 13349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:18.481420 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "616aba1e234b4a65a171c9c0fdff8efd"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "26c291450a002f86f9e7581245b0998b"
server_key_iv: "69eb1ab171d22c792ee5d99208a9df67"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.481498 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:18.509166 13349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:18.509501 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:18.509662 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:18.513559 13349 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:36683
I20260925 19:13:18.513599 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:36683 every 8 connection(s)
I20260925 19:13:18.513971 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:18.514986 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.517233 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: Bootstrap starting.
I20260925 19:13:18.517755 13416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.517988 13416 log.cc:824] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.518630 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: No bootstrap required, opened a new log
I20260925 19:13:18.520262 13416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } }
I20260925 19:13:18.520397 13416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.520419 13416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616aba1e234b4a65a171c9c0fdff8efd, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.520550 13416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } }
I20260925 19:13:18.520628 13416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:18.520651 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:18.520685 13416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:18.521142 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } }
I20260925 19:13:18.521258 13416 leader_election.cc:302] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616aba1e234b4a65a171c9c0fdff8efd; no voters:
I20260925 19:13:18.521399 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13349
I20260925 19:13:18.521438 13416 leader_election.cc:288] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:18.521507 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:18.521512 13419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:18.521663 13419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [term 1 LEADER]: Becoming Leader. State: Replica: 616aba1e234b4a65a171c9c0fdff8efd, State: Running, Role: LEADER
I20260925 19:13:18.521687 13416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:18.521733 28345 external_mini_cluster.cc:1428] Setting key 0ce8bb6f202a05acd3cd72386f9ab3a1
I20260925 19:13:18.522554 13419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } }
I20260925 19:13:18.523118 13420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616aba1e234b4a65a171c9c0fdff8efd. Latest consensus state: current_term: 1 leader_uuid: "616aba1e234b4a65a171c9c0fdff8efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } } }
I20260925 19:13:18.523257 13421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616aba1e234b4a65a171c9c0fdff8efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616aba1e234b4a65a171c9c0fdff8efd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36683 } } }
I20260925 19:13:18.523380 13421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:18.523365 13420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd [sys.catalog]: This master's current role is: LEADER
W20260925 19:13:18.523726 13435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:18.523787 13435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:18.523862 13438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:18.524019 13438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:18.525584 13438 catalog_manager.cc:1347] Generated new cluster ID: f2c29b32148f4b2985bb04d307fe26fd
I20260925 19:13:18.525638 13438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:18.528210 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.522221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.27.174.126:36683)
0925 19:13:18.522476 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.522486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.522519 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:13:18.522631 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.522638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.522704 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.523859 (+ 1155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.523953 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.525318 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.526209 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.526637 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.526777 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.527488 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.527492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.527492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:18.527725 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.527760 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:18.527771 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.527838 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.527926 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.528004 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260925 19:13:18.544171 13438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:18.544795 13438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:18.551589 13438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: Generated new TSK 0
I20260925 19:13:18.551841 13438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:18.558517 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--builtin_ntp_servers=127.27.174.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:18.651739 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:18.651964 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:18.652001 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:18.653203 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:18.653268 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:18.653313 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:18.655049 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:18.655378 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:18.655747 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:18.660943 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.660943 13446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.661274 13445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.661620 13440 server_base.cc:1034] running on GCE node
I20260925 19:13:18.661844 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:18.662107 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:18.663286 13440 hybrid_clock.cc:648] HybridClock initialized: now 1790363598663272 us; error 26 us; skew 500 ppm
I20260925 19:13:18.664758 13440 webserver.cc:460] Webserver started at http://127.27.174.65:43205/ using document root <none> and password file <none>
I20260925 19:13:18.665030 13440 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:18.665088 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:18.665225 13440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:18.666633 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "18c29b2f4db4439f9b307e4906dbbd24"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "8bb7b7dc905717026b4ffb12afae875c"
server_key_iv: "b261c0846d5479a9143d08d6fe690905"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.667083 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "18c29b2f4db4439f9b307e4906dbbd24"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "8bb7b7dc905717026b4ffb12afae875c"
server_key_iv: "b261c0846d5479a9143d08d6fe690905"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.668562 13440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 19:13:18.669339 13454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.669528 13440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:18.669610 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "18c29b2f4db4439f9b307e4906dbbd24"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "8bb7b7dc905717026b4ffb12afae875c"
server_key_iv: "b261c0846d5479a9143d08d6fe690905"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.669667 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:18.676399 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:18.676636 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:18.676731 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:18.676910 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:18.677151 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:18.677181 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.677201 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:18.677213 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.682703 13440 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:42459
I20260925 19:13:18.682770 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:42459 every 8 connection(s)
I20260925 19:13:18.683094 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:18.683876 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13440
I20260925 19:13:18.683967 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:18.684207 28345 external_mini_cluster.cc:1428] Setting key a19d9df6ba7d3d284165d1388584ad76
I20260925 19:13:18.685323 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--builtin_ntp_servers=127.27.174.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:13:18.689066 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.683892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:44307 (local address 127.27.174.126:36683)
0925 19:13:18.683958 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.683961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.684246 (+ 285us) server_negotiation.cc:407] Connection header received
0925 19:13:18.684921 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.684923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.684935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.685011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.685908 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.686488 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.687551 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.687807 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.687861 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.688831 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.688832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.688833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:18.688839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.688853 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:18.688857 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.688904 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.688981 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.689007 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:13:18.689375 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.683881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36683 (local address 127.27.174.65:44307)
0925 19:13:18.684118 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.684165 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.684793 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.685091 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.685095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.685105 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:18.685817 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.685824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.686596 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.686597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.687462 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.687465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.687881 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.687882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.688047 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.688667 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:18.688667 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:18.688709 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:18.688748 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:18.688905 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:18.688909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:18.688912 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.688986 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.689104 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260925 19:13:18.689946 13568 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36683
I20260925 19:13:18.690063 13568 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:18.690277 13568 heartbeater.cc:511] Master 127.27.174.126:36683 requested a full tablet report, sending...
I20260925 19:13:18.690801 13380 ts_manager.cc:194] Registered new tserver with Master: 18c29b2f4db4439f9b307e4906dbbd24 (127.27.174.65:42459)
I20260925 19:13:18.691416 13380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:44307
W20260925 19:13:18.765514 13571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:18.765707 13571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:18.765728 13571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:18.766878 13571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:18.766918 13571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:18.766942 13571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:18.768329 13571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:18.768564 13571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:18.768851 13571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:18.773833 13579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.773833 13577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.773928 13576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.774034 13571 server_base.cc:1034] running on GCE node
I20260925 19:13:18.774297 13571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:18.774537 13571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:18.775686 13571 hybrid_clock.cc:648] HybridClock initialized: now 1790363598775670 us; error 40 us; skew 500 ppm
I20260925 19:13:18.776952 13571 webserver.cc:460] Webserver started at http://127.27.174.66:39669/ using document root <none> and password file <none>
I20260925 19:13:18.777208 13571 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:18.777275 13571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:18.777410 13571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:18.778452 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "b5eada8654f54ed584fdd53fdd74e26f"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "f6e1f0ccbe9514dc788149aac46aff46"
server_key_iv: "58204993db57b4f46d18b5e548876ae8"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.778805 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "b5eada8654f54ed584fdd53fdd74e26f"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "f6e1f0ccbe9514dc788149aac46aff46"
server_key_iv: "58204993db57b4f46d18b5e548876ae8"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.780166 13571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:18.780836 13585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.780995 13571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:18.781086 13571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "b5eada8654f54ed584fdd53fdd74e26f"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "f6e1f0ccbe9514dc788149aac46aff46"
server_key_iv: "58204993db57b4f46d18b5e548876ae8"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.781164 13571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:18.788329 13571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:18.788609 13571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:18.788748 13571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:18.788954 13571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:18.789265 13571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:18.789315 13571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.789355 13571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:18.789383 13571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.795043 13571 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:42935
I20260925 19:13:18.795078 13698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:42935 every 8 connection(s)
I20260925 19:13:18.795409 13571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:13:18.800262 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13571
I20260925 19:13:18.800390 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:18.800603 28345 external_mini_cluster.cc:1428] Setting key dccbdae694bf3ef652ab6380ee40d56c
I20260925 19:13:18.800735 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.796174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:36679 (local address 127.27.174.126:36683)
0925 19:13:18.796243 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.796247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.796505 (+ 258us) server_negotiation.cc:407] Connection header received
0925 19:13:18.796972 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.796973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.796982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.797037 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.797683 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.798172 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.799137 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.799355 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.799409 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.800415 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.800417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.800417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:18.800422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.800438 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:18.800443 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.800495 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.800605 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.800632 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:13:18.800979 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.796172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36683 (local address 127.27.174.66:36679)
0925 19:13:18.796383 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.796417 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.796891 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.797109 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.797112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.797120 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:18.797608 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.797612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.798261 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.798262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.799062 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.799064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.799459 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.799461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.799664 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.800246 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:18.800247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:18.800276 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:18.800299 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:18.800504 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:18.800512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:18.800516 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.800603 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.800739 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260925 19:13:18.801395 13699 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36683
I20260925 19:13:18.801501 13699 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:18.801582 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--builtin_ntp_servers=127.27.174.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:13:18.801726 13699 heartbeater.cc:511] Master 127.27.174.126:36683 requested a full tablet report, sending...
I20260925 19:13:18.802100 13380 ts_manager.cc:194] Registered new tserver with Master: b5eada8654f54ed584fdd53fdd74e26f (127.27.174.66:42935)
I20260925 19:13:18.802591 13380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:36679
W20260925 19:13:18.881001 13702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:18.881201 13702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:18.881222 13702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:18.882264 13702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:18.882295 13702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:18.882318 13702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:18.883656 13702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:18.883834 13702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:18.884105 13702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:18.888962 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.888990 13710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:18.888962 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:18.889067 13702 server_base.cc:1034] running on GCE node
I20260925 19:13:18.889416 13702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:18.889650 13702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:18.890834 13702 hybrid_clock.cc:648] HybridClock initialized: now 1790363598890811 us; error 33 us; skew 500 ppm
I20260925 19:13:18.892074 13702 webserver.cc:460] Webserver started at http://127.27.174.67:38723/ using document root <none> and password file <none>
I20260925 19:13:18.892305 13702 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:18.892369 13702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:18.892511 13702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:18.893524 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "68a187689d61426fb3b3552a432c5aa2"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "2f82b71dcde71c29c90b858e694f923b"
server_key_iv: "9fc03d1dc95e379d52212bd9f667b397"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.893885 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "68a187689d61426fb3b3552a432c5aa2"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "2f82b71dcde71c29c90b858e694f923b"
server_key_iv: "9fc03d1dc95e379d52212bd9f667b397"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.895119 13702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:13:18.895823 13716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.895991 13702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:18.896088 13702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "68a187689d61426fb3b3552a432c5aa2"
format_stamp: "Formatted at 2026-09-25 19:13:18 on dist-test-slave-fcdn"
server_key: "2f82b71dcde71c29c90b858e694f923b"
server_key_iv: "9fc03d1dc95e379d52212bd9f667b397"
server_key_version: "encryptionkey@0"
I20260925 19:13:18.896150 13702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:18.915184 13702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:18.915521 13702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:18.915695 13702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:18.915920 13702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:18.916256 13702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:18.916288 13702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.916337 13702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:18.916368 13702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:18.921818 13702 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:35773
I20260925 19:13:18.921890 13829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:35773 every 8 connection(s)
I20260925 19:13:18.922204 13702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:18.926380 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13702
I20260925 19:13:18.926491 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:18.926688 28345 external_mini_cluster.cc:1428] Setting key 05a89d37e7cd3603e321afa44365b811
I20260925 19:13:18.927520 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.922955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40701 (local address 127.27.174.126:36683)
0925 19:13:18.923021 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.923023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.923284 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:13:18.923864 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.923866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.923878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.923929 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.924583 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.925087 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.926047 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.926275 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.926351 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.927254 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.927255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.927255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:18.927260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.927274 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:18.927278 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.927331 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.927411 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.927443 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:13:18.927737 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.922967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36683 (local address 127.27.174.67:40701)
0925 19:13:18.923166 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.923198 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.923762 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.924009 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.924012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.924020 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:18.924485 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.924489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.925178 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.925179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.925959 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.925961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.926357 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.926358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.926483 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.927102 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:18.927103 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:18.927154 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:18.927182 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:18.927335 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:18.927339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:18.927342 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.927411 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.927507 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 19:13:18.928102 13830 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36683
I20260925 19:13:18.928203 13830 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:18.928372 13830 heartbeater.cc:511] Master 127.27.174.126:36683 requested a full tablet report, sending...
I20260925 19:13:18.928663 13380 ts_manager.cc:194] Registered new tserver with Master: 68a187689d61426fb3b3552a432c5aa2 (127.27.174.67:35773)
I20260925 19:13:18.928984 13380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:40701
I20260925 19:13:18.938498 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:18.942692 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.939876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.27.174.126:36683)
0925 19:13:18.939953 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.939955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.940041 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:13:18.940104 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.940105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.940117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.940165 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:18.940362 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.940829 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.941561 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.941769 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.941840 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.942452 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:18.942453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:18.942454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:18.942459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:18.942472 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:18.942476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:18.942519 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.942582 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.942606 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 19:13:18.944271 13380 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:18.945791 13380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:18.951589 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.947719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:42459 (local address 127.0.0.1:36906)
0925 19:13:18.947913 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.947953 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.948017 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.948238 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.948241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.948243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.948310 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.948312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949368 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.949369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.950530 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.950532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951315 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.951387 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951400 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.951461 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.951515 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260925 19:13:18.951637 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.947893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.27.174.65:42459)
0925 19:13:18.948060 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.948065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.948076 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:18.948119 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.948121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.948135 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.948226 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.948377 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949288 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.950624 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951221 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951312 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951402 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.951492 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.951531 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260925 19:13:18.952023 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.947725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:42935 (local address 127.0.0.1:35654)
0925 19:13:18.948071 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.948081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.948119 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.948415 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.948415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.948417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.948450 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.948451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949773 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.949773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.950821 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.950823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951779 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.951840 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951844 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.951901 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.951938 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":100,"threads_started":1}
I20260925 19:13:18.952055 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.947821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.27.174.66:42935)
0925 19:13:18.948251 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.948256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.948265 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:18.948300 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.948303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.948318 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.948520 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.948583 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949690 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.950897 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951585 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951667 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951746 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.951920 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.951969 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":74,"threads_started":1}
I20260925 19:13:18.952215 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.948138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.27.174.67:35773)
0925 19:13:18.948451 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:18.948455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:18.948464 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:18.948502 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:18.948504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:18.948539 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:18.948613 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:18.948785 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949810 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.950778 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951492 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951561 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951680 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:18.952070 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 19:13:18.952121 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":77,"threads_started":1}
I20260925 19:13:18.952556 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:18.947947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35773 (local address 127.0.0.1:48732)
0925 19:13:18.948303 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:18.948312 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:18.948355 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:18.948675 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:18.948676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:18.948678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:18.948710 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.948711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.949883 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.949884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.950691 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:18.950694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:18.951891 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:18.951892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:18.951943 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:18.951946 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:18.951986 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 19:13:18.952015 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":67,"threads_started":1}
I20260925 19:13:18.952487 13502 tablet_service.cc:1467] Processing CreateTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 (DEFAULT_TABLE table=test-table [id=42660b76650e4153aa8a11749c6b0075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.952864 13502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.953126 13633 tablet_service.cc:1467] Processing CreateTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 (DEFAULT_TABLE table=test-table [id=42660b76650e4153aa8a11749c6b0075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.953388 13633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.955443 13844 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Bootstrap starting.
I20260925 19:13:18.955801 13845 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Bootstrap starting.
I20260925 19:13:18.956094 13764 tablet_service.cc:1467] Processing CreateTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 (DEFAULT_TABLE table=test-table [id=42660b76650e4153aa8a11749c6b0075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:18.956388 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:18.956418 13844 tablet_bootstrap.cc:654] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.956435 13845 tablet_bootstrap.cc:654] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.956765 13844 log.cc:824] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.956804 13845 log.cc:824] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.957563 13845 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: No bootstrap required, opened a new log
I20260925 19:13:18.957635 13845 ts_tablet_manager.cc:1397] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:18.957787 13844 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: No bootstrap required, opened a new log
I20260925 19:13:18.957849 13844 ts_tablet_manager.cc:1397] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:18.959753 13848 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Bootstrap starting.
I20260925 19:13:18.960273 13848 tablet_bootstrap.cc:654] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:18.960304 13845 raft_consensus.cc:348] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.960476 13845 raft_consensus.cc:374] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.960546 13845 raft_consensus.cc:729] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5eada8654f54ed584fdd53fdd74e26f, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.960598 13848 log.cc:824] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:18.960546 13844 raft_consensus.cc:348] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.960700 13844 raft_consensus.cc:374] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.960672 13845 consensus_queue.cc:260] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.960779 13844 raft_consensus.cc:729] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c29b2f4db4439f9b307e4906dbbd24, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.960935 13845 ts_tablet_manager.cc:1428] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:18.960908 13844 consensus_queue.cc:260] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.960976 13699 heartbeater.cc:503] Master 127.27.174.126:36683 was elected leader, sending a full tablet report...
I20260925 19:13:18.961148 13844 ts_tablet_manager.cc:1428] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:18.961172 13568 heartbeater.cc:503] Master 127.27.174.126:36683 was elected leader, sending a full tablet report...
I20260925 19:13:18.961905 13848 tablet_bootstrap.cc:492] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: No bootstrap required, opened a new log
I20260925 19:13:18.961987 13848 ts_tablet_manager.cc:1397] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:18.963891 13848 raft_consensus.cc:348] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.964030 13848 raft_consensus.cc:374] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:18.964072 13848 raft_consensus.cc:729] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a187689d61426fb3b3552a432c5aa2, State: Initialized, Role: FOLLOWER
I20260925 19:13:18.964178 13848 consensus_queue.cc:260] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:18.964408 13848 ts_tablet_manager.cc:1428] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 19:13:18.964480 13830 heartbeater.cc:503] Master 127.27.174.126:36683 was elected leader, sending a full tablet report...
W20260925 19:13:19.046012 13700 tablet.cc:2369] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:19.169466 13850 raft_consensus.cc:482] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:19.169615 13850 raft_consensus.cc:504] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:19.169978 13850 leader_election.cc:288] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c29b2f4db4439f9b307e4906dbbd24 (127.27.174.65:42459), 68a187689d61426fb3b3552a432c5aa2 (127.27.174.67:35773)
W20260925 19:13:19.172709 13831 tablet.cc:2369] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:19.173730 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.170469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39115 (local address 127.27.174.67:35773)
0925 19:13:19.170527 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:19.170533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:19.170595 (+ 62us) server_negotiation.cc:407] Connection header received
0925 19:13:19.170660 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:19.170661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:19.170668 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:19.170716 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:19.170862 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.171537 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.172972 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.173374 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.173459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:19.173468 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:19.173619 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 19:13:19.173661 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:13:19.173750 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.170342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35773 (local address 127.27.174.66:39115)
0925 19:13:19.170512 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:19.170521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:19.170597 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:19.170739 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:19.170740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:19.170742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:19.170805 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:19.170806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.171636 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.171637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:19.172885 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:19.172888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.173461 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.173462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:19.173545 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:19.173549 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:19.173605 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:13:19.173653 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260925 19:13:19.173792 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.170244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:55681 (local address 127.27.174.65:42459)
0925 19:13:19.170298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:19.170301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:19.170366 (+ 65us) server_negotiation.cc:407] Connection header received
0925 19:13:19.170463 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:19.170464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:19.170477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:19.170526 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:19.170773 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.171537 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.173028 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.173395 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.173492 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:19.173501 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:19.173673 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 19:13:19.173719 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:19.173758 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.170173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:42459 (local address 127.27.174.66:55681)
0925 19:13:19.170263 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:19.170282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:19.170378 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:19.170594 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:19.170595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:19.170596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:19.170672 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:19.170675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.171646 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.171647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:19.172885 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:19.172888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.173546 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.173547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:19.173617 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:19.173619 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:13:19.173667 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:13:19.173697 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":2}
I20260925 19:13:19.174074 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e4c8e8cdbb431bbcdabb7516e5cb48" candidate_uuid: "b5eada8654f54ed584fdd53fdd74e26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a187689d61426fb3b3552a432c5aa2" is_pre_election: true
I20260925 19:13:19.174222 13784 raft_consensus.cc:2454] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5eada8654f54ed584fdd53fdd74e26f in term 0.
I20260925 19:13:19.174316 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e4c8e8cdbb431bbcdabb7516e5cb48" candidate_uuid: "b5eada8654f54ed584fdd53fdd74e26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c29b2f4db4439f9b307e4906dbbd24" is_pre_election: true
I20260925 19:13:19.174448 13587 leader_election.cc:302] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a187689d61426fb3b3552a432c5aa2, b5eada8654f54ed584fdd53fdd74e26f; no voters:
I20260925 19:13:19.174481 13522 raft_consensus.cc:2454] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5eada8654f54ed584fdd53fdd74e26f in term 0.
I20260925 19:13:19.174575 13850 raft_consensus.cc:2781] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:19.174638 13850 raft_consensus.cc:482] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:19.174665 13850 raft_consensus.cc:3037] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:19.175361 13850 raft_consensus.cc:504] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:19.175457 13850 leader_election.cc:288] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [CANDIDATE]: Term 1 election: Requested vote from peers 18c29b2f4db4439f9b307e4906dbbd24 (127.27.174.65:42459), 68a187689d61426fb3b3552a432c5aa2 (127.27.174.67:35773)
I20260925 19:13:19.175642 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e4c8e8cdbb431bbcdabb7516e5cb48" candidate_uuid: "b5eada8654f54ed584fdd53fdd74e26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c29b2f4db4439f9b307e4906dbbd24"
I20260925 19:13:19.175693 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e4c8e8cdbb431bbcdabb7516e5cb48" candidate_uuid: "b5eada8654f54ed584fdd53fdd74e26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a187689d61426fb3b3552a432c5aa2"
I20260925 19:13:19.175717 13522 raft_consensus.cc:3037] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:19.175788 13784 raft_consensus.cc:3037] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:19.176468 13522 raft_consensus.cc:2454] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5eada8654f54ed584fdd53fdd74e26f in term 1.
I20260925 19:13:19.176568 13784 raft_consensus.cc:2454] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5eada8654f54ed584fdd53fdd74e26f in term 1.
I20260925 19:13:19.176625 13587 leader_election.cc:302] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c29b2f4db4439f9b307e4906dbbd24, b5eada8654f54ed584fdd53fdd74e26f; no voters:
I20260925 19:13:19.176704 13850 raft_consensus.cc:2781] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:19.176837 13850 raft_consensus.cc:686] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 1 LEADER]: Becoming Leader. State: Replica: b5eada8654f54ed584fdd53fdd74e26f, State: Running, Role: LEADER
I20260925 19:13:19.176929 13850 consensus_queue.cc:237] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } }
I20260925 19:13:19.177810 13379 catalog_manager.cc:5495] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f reported cstate change: term changed from 0 to 1, leader changed from <none> to b5eada8654f54ed584fdd53fdd74e26f (127.27.174.66). New cstate: current_term: 1 leader_uuid: "b5eada8654f54ed584fdd53fdd74e26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5eada8654f54ed584fdd53fdd74e26f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 } health_report { overall_health: UNKNOWN } } }
W20260925 19:13:19.183791 13569 tablet.cc:2369] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:19.232767 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.230535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.27.174.66:42935)
0925 19:13:19.230595 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:19.230597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:19.230605 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:19.230641 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:19.230642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:19.230659 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:19.230716 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:19.230828 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.231282 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.232070 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:19.232248 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:19.232297 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:19.232441 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:19.232534 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:19.232592 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:19.232634 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 19:13:19.232649 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:13:19.234581 13784 raft_consensus.cc:1261] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 1 FOLLOWER]: Refusing update from remote peer b5eada8654f54ed584fdd53fdd74e26f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:19.234581 13522 raft_consensus.cc:1261] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 1 FOLLOWER]: Refusing update from remote peer b5eada8654f54ed584fdd53fdd74e26f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:19.234920 13850 consensus_queue.cc:1035] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a187689d61426fb3b3552a432c5aa2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:19.235073 13854 consensus_queue.cc:1035] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c29b2f4db4439f9b307e4906dbbd24" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:19.237459 13858 mvcc.cc:205] Tried to move back new op lower bound from 7333329302462083072 to 7333329302229487616. Current Snapshot: MvccSnapshot[applied={T|T < 7333329302462083072}]
I20260925 19:13:19.238407 13859 mvcc.cc:205] Tried to move back new op lower bound from 7333329302462083072 to 7333329302229487616. Current Snapshot: MvccSnapshot[applied={T|T < 7333329302462083072 or (T in {7333329302462083072})}]
I20260925 19:13:19.239686 13379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:19.239779 13379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:19.240451 13379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 616aba1e234b4a65a171c9c0fdff8efd: Sending DeleteTablet for 3 replicas of tablet 79e4c8e8cdbb431bbcdabb7516e5cb48
I20260925 19:13:19.240783 13502 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:19 UTC) from {username='slave'} at 127.0.0.1:36906
I20260925 19:13:19.240927 13633 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:19 UTC) from {username='slave'} at 127.0.0.1:35654
I20260925 19:13:19.240993 13866 tablet_replica.cc:330] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: stopping tablet replica
I20260925 19:13:19.240998 13764 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:19 UTC) from {username='slave'} at 127.0.0.1:48732
I20260925 19:13:19.241081 13866 raft_consensus.cc:2229] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:19.241181 13866 raft_consensus.cc:2258] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:19.241369 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13440
I20260925 19:13:19.241597 13866 ts_tablet_manager.cc:1905] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 18c29b2f4db4439f9b307e4906dbbd24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:19.241325 13868 tablet_replica.cc:330] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: stopping tablet replica
I20260925 19:13:19.241994 13868 raft_consensus.cc:2229] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:19.242066 13868 raft_consensus.cc:2258] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:19.242411 13868 ts_tablet_manager.cc:1905] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:19.243755 13868 ts_tablet_manager.cc:1918] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:19.243819 13868 log.cc:1196] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/79e4c8e8cdbb431bbcdabb7516e5cb48
I20260925 19:13:19.244078 13868 ts_tablet_manager.cc:1939] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P 68a187689d61426fb3b3552a432c5aa2: Deleting consensus metadata
I20260925 19:13:19.244979 13365 catalog_manager.cc:4841] TS 68a187689d61426fb3b3552a432c5aa2 (127.27.174.67:35773): tablet 79e4c8e8cdbb431bbcdabb7516e5cb48 (table test-table [id=42660b76650e4153aa8a11749c6b0075]) successfully deleted
W20260925 19:13:19.247635 13365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:42459 (error 108)
I20260925 19:13:19.247753 13867 tablet_replica.cc:330] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f: stopping tablet replica
I20260925 19:13:19.247825 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13571
I20260925 19:13:19.247848 13867 raft_consensus.cc:2229] T 79e4c8e8cdbb431bbcdabb7516e5cb48 P b5eada8654f54ed584fdd53fdd74e26f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:13:19.249119 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.248984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:42459 (local address 127.0.0.1:36908)
0925 19:13:19.249039 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:19.249084 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:13:19.249226 13365 catalog_manager.cc:4584] TS 18c29b2f4db4439f9b307e4906dbbd24 (127.27.174.65:42459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79e4c8e8cdbb431bbcdabb7516e5cb48: Network error: Client connection negotiation failed: client connection to 127.27.174.65:42459: connect: Connection refused (error 111)
I20260925 19:13:19.253260 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:19.253160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:42935 (local address 127.0.0.1:35680)
0925 19:13:19.253210 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:19.253236 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.66:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:13:19.253263 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13702
I20260925 19:13:19.257653 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13349
2026-09-25T19:13:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (896 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:19 dist-test-slave-fcdn krb5kdc[13872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:19 dist-test-slave-fcdn krb5kdc[13872](info): set up 2 sockets
Sep 25 19:13:19 dist-test-slave-fcdn krb5kdc[13872](info): commencing operation
krb5kdc: starting...
W20260925 19:13:20.287372 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:20.402021 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41953
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:20.481406 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:20.481678 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:20.481725 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:20.483282 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:20.483331 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:20.483354 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:20.483384 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:20.483413 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:20.485615 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:20.485865 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:20.486100 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:20.491091 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.491084 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.491093 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.491364 13885 server_base.cc:1034] running on GCE node
I20260925 19:13:20.491580 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:20.491881 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:20.493041 13885 hybrid_clock.cc:648] HybridClock initialized: now 1790363600493030 us; error 35 us; skew 500 ppm
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:20.495376 13885 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:20.495878 13885 webserver.cc:460] Webserver started at http://127.27.174.126:36031/ using document root <none> and password file <none>
I20260925 19:13:20.496091 13885 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:20.496165 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:20.496304 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:20.497341 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "46149a418bb84977b6df3f8ce3823edd"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "0b76bd893d2f63bb934664a89acc01b6"
server_key_iv: "f9e1b4549b1585d65041392ab7af6f7b"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.497694 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "46149a418bb84977b6df3f8ce3823edd"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "0b76bd893d2f63bb934664a89acc01b6"
server_key_iv: "f9e1b4549b1585d65041392ab7af6f7b"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.498968 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:20.499653 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.499823 13885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:20.499914 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "46149a418bb84977b6df3f8ce3823edd"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "0b76bd893d2f63bb934664a89acc01b6"
server_key_iv: "f9e1b4549b1585d65041392ab7af6f7b"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.499991 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:20.512684 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:20.514899 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:20.515048 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:20.518898 13885 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41953
I20260925 19:13:20.518953 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41953 every 8 connection(s)
I20260925 19:13:20.519271 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:20.520153 13953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:20.522336 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: Bootstrap starting.
I20260925 19:13:20.522883 13953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:20.523159 13953 log.cc:824] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:20.523862 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: No bootstrap required, opened a new log
I20260925 19:13:20.525388 13953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } }
I20260925 19:13:20.525517 13953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:20.525560 13953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46149a418bb84977b6df3f8ce3823edd, State: Initialized, Role: FOLLOWER
I20260925 19:13:20.525723 13953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } }
I20260925 19:13:20.525806 13953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:20.525856 13953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:20.525903 13953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:20.526023 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13885
I20260925 19:13:20.526124 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:20.526254 28345 external_mini_cluster.cc:1428] Setting key 215c97a317054991b96c4e82b0e62b9c
I20260925 19:13:20.526381 13953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } }
I20260925 19:13:20.526479 13953 leader_election.cc:302] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46149a418bb84977b6df3f8ce3823edd; no voters:
I20260925 19:13:20.526647 13953 leader_election.cc:288] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:20.526751 13956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:20.526896 13953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:20.526924 13956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [term 1 LEADER]: Becoming Leader. State: Replica: 46149a418bb84977b6df3f8ce3823edd, State: Running, Role: LEADER
I20260925 19:13:20.527135 13956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } }
I20260925 19:13:20.527639 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46149a418bb84977b6df3f8ce3823edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } } }
I20260925 19:13:20.527747 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:20.527882 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46149a418bb84977b6df3f8ce3823edd. Latest consensus state: current_term: 1 leader_uuid: "46149a418bb84977b6df3f8ce3823edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46149a418bb84977b6df3f8ce3823edd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41953 } } }
I20260925 19:13:20.527951 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:20.528081 13967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:20.528584 13967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:20.530187 13967 catalog_manager.cc:1347] Generated new cluster ID: b1cbe9904c5848a5b6df8e9ba9817124
I20260925 19:13:20.530239 13967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:20.538152 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:20.526750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.27.174.126:41953)
0925 19:13:20.526911 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:20.526920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:20.527101 (+ 181us) server_negotiation.cc:407] Connection header received
0925 19:13:20.527260 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:20.527267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:20.527327 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:20.527478 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:20.528338 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.529578 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.530634 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.531221 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.531317 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:20.534052 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:20.534056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:20.534056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:20.534068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:20.536591 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.536959 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.536962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.536962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.537079 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.537274 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.537275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.537276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.537575 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:20.537739 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:20.537806 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:13:20.537938 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260925 19:13:20.556425 13967 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:20.557200 13967 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:20.569759 13967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: Generated new TSK 0
I20260925 19:13:20.569957 13967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:13:20.596437 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:20.677203 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:20.677393 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:20.677415 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:20.678475 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:20.678511 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:20.678537 13979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:20.679965 13979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:20.680147 13979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:20.680322 13979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:20.685364 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.685418 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.685401 13979 server_base.cc:1034] running on GCE node
W20260925 19:13:20.685385 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.685803 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:20.686051 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:20.687206 13979 hybrid_clock.cc:648] HybridClock initialized: now 1790363600687205 us; error 36 us; skew 500 ppm
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:20.689620 13979 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:13:20.690042 13979 webserver.cc:460] Webserver started at http://127.27.174.65:39095/ using document root <none> and password file <none>
I20260925 19:13:20.690270 13979 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:20.690333 13979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:20.690474 13979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:20.691470 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "afe71c4b19e648b8b55f34e7d7dce36f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "69eaa7f2c24fe3cf2cb0bacde801d18e"
server_key_iv: "188f55ae78131d3180bee4971cf3269d"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.691828 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "afe71c4b19e648b8b55f34e7d7dce36f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "69eaa7f2c24fe3cf2cb0bacde801d18e"
server_key_iv: "188f55ae78131d3180bee4971cf3269d"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.693118 13979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:20.693868 13994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.694049 13979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:20.694151 13979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "afe71c4b19e648b8b55f34e7d7dce36f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "69eaa7f2c24fe3cf2cb0bacde801d18e"
server_key_iv: "188f55ae78131d3180bee4971cf3269d"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.694231 13979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:20.708906 13979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:20.710666 13979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:20.710814 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:20.711024 13979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:20.711314 13979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:20.711366 13979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.711406 13979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:20.711434 13979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.717296 13979 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38275
I20260925 19:13:20.717340 14107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38275 every 8 connection(s)
I20260925 19:13:20.717631 13979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:20.721451 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 13979
I20260925 19:13:20.721535 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:20.721741 28345 external_mini_cluster.cc:1428] Setting key 43c08dd8e865c9e5069a90e7c22bfba4
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:20.727190 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:20.718272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:41143 (local address 127.27.174.126:41953)
0925 19:13:20.718334 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:20.718337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:20.718581 (+ 244us) server_negotiation.cc:407] Connection header received
0925 19:13:20.719084 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:20.719085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:20.719095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:20.719144 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:20.720157 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.720653 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.721723 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.721999 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.722091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:20.724365 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:20.724366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:20.724367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:20.724376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:20.725784 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.726269 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.726270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.726270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.726320 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.726602 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.726602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.726603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.726698 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:20.726742 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:20.727064 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 19:13:20.727126 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:13:20.727509 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:20.718258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41953 (local address 127.27.174.65:41143)
0925 19:13:20.718439 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:20.718492 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:20.718962 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:20.719210 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:20.719213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:20.719866 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:20.720063 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:20.720068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.720742 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.720743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:20.721623 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:20.721627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.722102 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.722103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:20.722244 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:20.722698 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:20.722707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:20.724279 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:20.725862 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:20.725869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:20.725871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:20.726197 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:20.726385 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:20.726386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:20.726387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:20.726535 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:20.726755 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:20.726762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:20.726923 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 19:13:20.727049 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 19:13:20.727226 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260925 19:13:20.728039 14108 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41953
I20260925 19:13:20.728125 14108 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:20.728312 14108 heartbeater.cc:511] Master 127.27.174.126:41953 requested a full tablet report, sending...
I20260925 19:13:20.728766 13917 ts_manager.cc:194] Registered new tserver with Master: afe71c4b19e648b8b55f34e7d7dce36f (127.27.174.65:38275)
I20260925 19:13:20.729336 13917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:41143
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:13:20.746707 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:20.825872 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:20.826105 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:20.826145 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:20.827409 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:20.827464 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:20.827508 14113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:20.829067 14113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:20.829327 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:20.829555 14113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:20.834337 14119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.834394 14121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.834342 14118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.834503 14113 server_base.cc:1034] running on GCE node
I20260925 19:13:20.834694 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:20.834921 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:20.836068 14113 hybrid_clock.cc:648] HybridClock initialized: now 1790363600836046 us; error 31 us; skew 500 ppm
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:20.838511 14113 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:13:20.838932 14113 webserver.cc:460] Webserver started at http://127.27.174.66:35309/ using document root <none> and password file <none>
I20260925 19:13:20.839110 14113 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:20.839149 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:20.839233 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:20.840233 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "780aa75b3bcc4224b433075d43ef2743"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "eda26bae01fd6b23bc73eb98c2df604d"
server_key_iv: "255e35e59d75a471ed1cc354c161e097"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.840521 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "780aa75b3bcc4224b433075d43ef2743"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "eda26bae01fd6b23bc73eb98c2df604d"
server_key_iv: "255e35e59d75a471ed1cc354c161e097"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.841681 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:20.842303 14128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.842471 14113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:20.842546 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "780aa75b3bcc4224b433075d43ef2743"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "eda26bae01fd6b23bc73eb98c2df604d"
server_key_iv: "255e35e59d75a471ed1cc354c161e097"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.842595 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:20.854722 14113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:20.856552 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:20.856654 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:20.856820 14113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:20.857043 14113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:20.857070 14113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.857090 14113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:20.857102 14113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.862358 14113 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:37643
I20260925 19:13:20.862448 14241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:37643 every 8 connection(s)
I20260925 19:13:20.862740 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:20.871007 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14113
I20260925 19:13:20.871116 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:20.871316 28345 external_mini_cluster.cc:1428] Setting key c78841842bd741099659c1b2e8f54a67
I20260925 19:13:20.871376 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:20.863420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:36257 (local address 127.27.174.126:41953)
0925 19:13:20.863468 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:20.863470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:20.863658 (+ 188us) server_negotiation.cc:407] Connection header received
0925 19:13:20.864133 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:20.864135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:20.864142 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:20.864193 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:20.865071 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.865694 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.866650 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.866860 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.866924 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:20.868865 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:20.868867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:20.868867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:20.868873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:20.870195 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.870506 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.870507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.870508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.870556 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:20.870743 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:20.870744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:20.870744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:20.870836 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:20.870884 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:20.871227 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 19:13:20.871298 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:13:20.871582 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:20.863396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41953 (local address 127.27.174.66:36257)
0925 19:13:20.863532 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:20.863571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:20.864043 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:20.864250 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:20.864252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:20.864781 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:20.864958 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:20.864962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.865786 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.865787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:20.866561 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:20.866563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:20.866934 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:20.866935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:20.867025 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:20.867424 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:20.867432 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:20.868785 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:20.870257 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:20.870262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:20.870263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:20.870447 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:20.870607 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:20.870608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:20.870608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:20.870694 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:20.870895 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:20.870900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:20.871064 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 19:13:20.871198 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 19:13:20.871362 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260925 19:13:20.872046 14242 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41953
I20260925 19:13:20.872139 14242 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:20.872300 14242 heartbeater.cc:511] Master 127.27.174.126:41953 requested a full tablet report, sending...
I20260925 19:13:20.872746 13916 ts_manager.cc:194] Registered new tserver with Master: 780aa75b3bcc4224b433075d43ef2743 (127.27.174.66:37643)
I20260925 19:13:20.873111 13916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:36257
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:13:20.894691 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:20.973670 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:20.973867 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:20.973889 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:20.974990 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:20.975029 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:20.975075 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:20.976553 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:20.976780 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:20.976987 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:20.981796 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.981860 14247 server_base.cc:1034] running on GCE node
W20260925 19:13:20.981796 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:20.981810 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:20.982239 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:20.982461 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:20.983647 14247 hybrid_clock.cc:648] HybridClock initialized: now 1790363600983634 us; error 38 us; skew 500 ppm
Sep 25 19:13:20 dist-test-slave-fcdn krb5kdc[13872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:20.985961 14247 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:13:20.986382 14247 webserver.cc:460] Webserver started at http://127.27.174.67:43427/ using document root <none> and password file <none>
I20260925 19:13:20.986599 14247 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:20.986662 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:20.986809 14247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:20.987854 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "79ed53dd5b6f4276b9e248464cbe6a0f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "f2114d4421e1d29d60bbcce5ae2e9443"
server_key_iv: "e3d2b9f3e1c86c19da5c02cf38ed0955"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.988199 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "79ed53dd5b6f4276b9e248464cbe6a0f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "f2114d4421e1d29d60bbcce5ae2e9443"
server_key_iv: "e3d2b9f3e1c86c19da5c02cf38ed0955"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.989403 14247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:13:20.990046 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.990211 14247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:20.990302 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "79ed53dd5b6f4276b9e248464cbe6a0f"
format_stamp: "Formatted at 2026-09-25 19:13:20 on dist-test-slave-fcdn"
server_key: "f2114d4421e1d29d60bbcce5ae2e9443"
server_key_iv: "e3d2b9f3e1c86c19da5c02cf38ed0955"
server_key_version: "encryptionkey@0"
I20260925 19:13:20.990373 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:21.006331 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:21.008128 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:21.008282 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:21.008487 14247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:21.008781 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:21.008836 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.008879 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:21.008914 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.014685 14247 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46253
I20260925 19:13:21.014743 14375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46253 every 8 connection(s)
I20260925 19:13:21.015023 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:21.019150 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14247
I20260925 19:13:21.019265 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:21.019464 28345 external_mini_cluster.cc:1428] Setting key d83b676e0bcbf8b74a91e6cf8404be69
Sep 25 19:13:21 dist-test-slave-fcdn krb5kdc[13872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363600, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:21.023931 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.015659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:47311 (local address 127.27.174.126:41953)
0925 19:13:21.015725 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.015727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.016002 (+ 275us) server_negotiation.cc:407] Connection header received
0925 19:13:21.016456 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.016457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.016467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.016517 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.017375 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.017852 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.018762 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.018934 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.018983 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.021561 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.021562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.021562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:21.021568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.022753 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:21.023096 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:21.023097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:21.023097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:21.023143 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:21.023337 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:21.023337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:21.023337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:21.023431 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.023478 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.023783 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.023865 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:13:21.024040 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.015688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41953 (local address 127.27.174.67:47311)
0925 19:13:21.015835 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.015905 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.016373 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.016565 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.016568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.017123 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:21.017302 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.017307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.017922 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.017923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.018691 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.018693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.019006 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.019007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.019154 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.019810 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:21.019820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:21.021477 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:21.022819 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:21.022825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:21.022826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:21.023034 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:21.023196 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:21.023197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:21.023197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:21.023288 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:21.023480 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:21.023484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:21.023590 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.023749 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.023866 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260925 19:13:21.024502 14376 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41953
I20260925 19:13:21.024581 14376 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:21.024718 14376 heartbeater.cc:511] Master 127.27.174.126:41953 requested a full tablet report, sending...
I20260925 19:13:21.025005 13916 ts_manager.cc:194] Registered new tserver with Master: 79ed53dd5b6f4276b9e248464cbe6a0f (127.27.174.67:46253)
I20260925 19:13:21.025379 13916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:47311
I20260925 19:13:21.031077 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:21.037701 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.032390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.27.174.126:41953)
0925 19:13:21.032426 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.032428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.032514 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:13:21.032563 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.032563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.032570 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.032627 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.033163 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.033618 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.034279 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.034446 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.034492 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.035666 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.035667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.035668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:21.035673 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.036781 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:21.037114 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:21.037115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:21.037115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:21.037161 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:21.037314 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:21.037315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:21.037315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:21.037418 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.037488 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.037603 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.037646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:13:21.039402 13916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:21.040939 13916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:21.047369 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.042993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38275 (local address 127.0.0.1:48866)
0925 19:13:21.043519 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.043529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.043576 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.043832 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.043833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.043835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.043888 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.043890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.044911 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.044913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.046203 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.046206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.047092 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.047093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.047146 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.047157 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.047241 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.047293 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260925 19:13:21.047381 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.043189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.27.174.65:38275)
0925 19:13:21.043492 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.043496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.043618 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:13:21.043665 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.043668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.043684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.043752 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.043966 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.044814 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.046283 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.047014 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.047091 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.047146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.047260 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.047305 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":101,"threads_started":1}
I20260925 19:13:21.047734 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.042987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37643 (local address 127.0.0.1:41320)
0925 19:13:21.043139 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.043176 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.043241 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.043988 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.043988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.043990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.044022 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.044023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.045274 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.045275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.046238 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.046240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.047474 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.047475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.047537 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.047541 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.047628 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.047666 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":49,"threads_started":1}
I20260925 19:13:21.047951 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.043136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.27.174.66:37643)
0925 19:13:21.043584 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.043588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.043616 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:13:21.043656 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.043658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.043685 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.043864 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.044130 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.045187 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.046342 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.047014 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.047091 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.047147 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.047828 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.047879 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":103,"threads_started":1}
I20260925 19:13:21.048121 14042 tablet_service.cc:1467] Processing CreateTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 (DEFAULT_TABLE table=test-table [id=86e98958d3d74b4f9a27529c7e94b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.048462 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb1bf2be5e04bc3873a54fc76642d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.048640 14176 tablet_service.cc:1467] Processing CreateTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 (DEFAULT_TABLE table=test-table [id=86e98958d3d74b4f9a27529c7e94b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.048794 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.042991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46253 (local address 127.0.0.1:57478)
0925 19:13:21.043367 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.043378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.043432 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.043738 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.043741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.043744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.043811 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.043814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.045095 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.045096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.046204 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.046206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.048539 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.048540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.048614 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.048617 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.048685 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.048725 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260925 19:13:21.048938 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb1bf2be5e04bc3873a54fc76642d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.049005 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.043220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.27.174.67:46253)
0925 19:13:21.043399 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.043404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.043417 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:13:21.043517 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.043520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.043532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.043713 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.043897 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.044996 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.047181 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.048391 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.048450 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.048506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.048700 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.048738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260925 19:13:21.049516 14310 tablet_service.cc:1467] Processing CreateTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 (DEFAULT_TABLE table=test-table [id=86e98958d3d74b4f9a27529c7e94b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.049794 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb1bf2be5e04bc3873a54fc76642d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.051875 14390 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: Bootstrap starting.
I20260925 19:13:21.052604 14390 tablet_bootstrap.cc:654] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.053089 14390 log.cc:824] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.053393 14391 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Bootstrap starting.
I20260925 19:13:21.053761 14390 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: No bootstrap required, opened a new log
I20260925 19:13:21.053815 14390 ts_tablet_manager.cc:1397] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:21.053956 14391 tablet_bootstrap.cc:654] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.054404 14391 log.cc:824] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.054808 14392 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Bootstrap starting.
I20260925 19:13:21.055274 14392 tablet_bootstrap.cc:654] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.055477 14390 raft_consensus.cc:348] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.055687 14390 raft_consensus.cc:374] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.055738 14390 raft_consensus.cc:729] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe71c4b19e648b8b55f34e7d7dce36f, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.055754 14392 log.cc:824] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.055857 14390 consensus_queue.cc:260] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.056102 14390 ts_tablet_manager.cc:1428] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:13:21.056171 14108 heartbeater.cc:503] Master 127.27.174.126:41953 was elected leader, sending a full tablet report...
I20260925 19:13:21.056169 14391 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: No bootstrap required, opened a new log
I20260925 19:13:21.056247 14391 ts_tablet_manager.cc:1397] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:13:21.057525 14391 raft_consensus.cc:348] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.057650 14391 raft_consensus.cc:374] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.057673 14391 raft_consensus.cc:729] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780aa75b3bcc4224b433075d43ef2743, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.057816 14391 consensus_queue.cc:260] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.058063 14391 ts_tablet_manager.cc:1428] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:21.058102 14242 heartbeater.cc:503] Master 127.27.174.126:41953 was elected leader, sending a full tablet report...
I20260925 19:13:21.058881 14392 tablet_bootstrap.cc:492] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: No bootstrap required, opened a new log
I20260925 19:13:21.059015 14392 ts_tablet_manager.cc:1397] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 19:13:21.060397 14392 raft_consensus.cc:348] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.060519 14392 raft_consensus.cc:374] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.060540 14392 raft_consensus.cc:729] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ed53dd5b6f4276b9e248464cbe6a0f, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.060664 14392 consensus_queue.cc:260] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.060899 14392 ts_tablet_manager.cc:1428] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:21.060945 14376 heartbeater.cc:503] Master 127.27.174.126:41953 was elected leader, sending a full tablet report...
W20260925 19:13:21.113263 14243 tablet.cc:2369] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:21.137214 14398 raft_consensus.cc:482] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:21.137387 14398 raft_consensus.cc:504] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.137748 14398 leader_election.cc:288] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780aa75b3bcc4224b433075d43ef2743 (127.27.174.66:37643), afe71c4b19e648b8b55f34e7d7dce36f (127.27.174.65:38275)
I20260925 19:13:21.140532 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.138042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:45483 (local address 127.27.174.65:38275)
0925 19:13:21.138104 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.138106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.138118 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:13:21.138148 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.138152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.138161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.138241 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.138393 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.138877 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.139815 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.140193 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.140273 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.140283 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.140425 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.140463 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:13:21.140533 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.137932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38275 (local address 127.27.174.67:45483)
0925 19:13:21.137992 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.138008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.138066 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.138246 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.138247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.138249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.138321 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.138323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.138960 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.138961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.139722 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.139724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.140280 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.140281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.140355 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.140358 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.140419 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.140460 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":18}
I20260925 19:13:21.140780 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.137933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37643 (local address 127.27.174.67:36765)
0925 19:13:21.138189 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.138196 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.138252 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.138388 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.138389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.138390 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.138461 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.138463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.139058 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.139059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.139839 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.139841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.140378 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.140379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.140445 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.140448 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.140504 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.140543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":14,"thread_start_us":116,"threads_started":1}
I20260925 19:13:21.140794 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.138081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:36765 (local address 127.27.174.66:37643)
0925 19:13:21.138127 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.138128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.138251 (+ 123us) server_negotiation.cc:407] Connection header received
0925 19:13:21.138304 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.138305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.138316 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.138384 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.138526 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.138979 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.139918 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.140291 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.140367 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.140375 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.140522 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.140557 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:13:21.140959 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb1bf2be5e04bc3873a54fc76642d88" candidate_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" is_pre_election: true
I20260925 19:13:21.141136 14062 raft_consensus.cc:2454] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ed53dd5b6f4276b9e248464cbe6a0f in term 0.
I20260925 19:13:21.141105 14196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb1bf2be5e04bc3873a54fc76642d88" candidate_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780aa75b3bcc4224b433075d43ef2743" is_pre_election: true
I20260925 19:13:21.141251 14196 raft_consensus.cc:2454] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ed53dd5b6f4276b9e248464cbe6a0f in term 0.
I20260925 19:13:21.141396 14263 leader_election.cc:302] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ed53dd5b6f4276b9e248464cbe6a0f, afe71c4b19e648b8b55f34e7d7dce36f; no voters:
I20260925 19:13:21.141566 14398 raft_consensus.cc:2781] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:21.141636 14398 raft_consensus.cc:482] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:21.141654 14398 raft_consensus.cc:3037] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.142385 14398 raft_consensus.cc:504] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.142524 14398 leader_election.cc:288] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [CANDIDATE]: Term 1 election: Requested vote from peers 780aa75b3bcc4224b433075d43ef2743 (127.27.174.66:37643), afe71c4b19e648b8b55f34e7d7dce36f (127.27.174.65:38275)
I20260925 19:13:21.142668 14196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb1bf2be5e04bc3873a54fc76642d88" candidate_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780aa75b3bcc4224b433075d43ef2743"
I20260925 19:13:21.142668 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb1bf2be5e04bc3873a54fc76642d88" candidate_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe71c4b19e648b8b55f34e7d7dce36f"
I20260925 19:13:21.142745 14062 raft_consensus.cc:3037] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.142745 14196 raft_consensus.cc:3037] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.143668 14062 raft_consensus.cc:2454] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ed53dd5b6f4276b9e248464cbe6a0f in term 1.
I20260925 19:13:21.143668 14196 raft_consensus.cc:2454] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ed53dd5b6f4276b9e248464cbe6a0f in term 1.
I20260925 19:13:21.143841 14266 leader_election.cc:302] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780aa75b3bcc4224b433075d43ef2743, 79ed53dd5b6f4276b9e248464cbe6a0f; no voters:
I20260925 19:13:21.143936 14398 raft_consensus.cc:2781] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:21.144070 14398 raft_consensus.cc:686] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 1 LEADER]: Becoming Leader. State: Replica: 79ed53dd5b6f4276b9e248464cbe6a0f, State: Running, Role: LEADER
I20260925 19:13:21.144179 14398 consensus_queue.cc:237] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } }
I20260925 19:13:21.145016 13917 catalog_manager.cc:5495] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ed53dd5b6f4276b9e248464cbe6a0f (127.27.174.67). New cstate: current_term: 1 leader_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ed53dd5b6f4276b9e248464cbe6a0f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 } health_report { overall_health: UNKNOWN } } }
I20260925 19:13:21.161574 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.159418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.27.174.67:46253)
0925 19:13:21.159471 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.159473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.159496 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:13:21.159525 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.159526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.159541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.159643 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:21.159767 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.160214 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.160884 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.161056 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.161106 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.161235 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:21.161332 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:21.161398 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.161436 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.161446 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:13:21.163254 14062 raft_consensus.cc:1261] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f [term 1 FOLLOWER]: Refusing update from remote peer 79ed53dd5b6f4276b9e248464cbe6a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:21.163257 14196 raft_consensus.cc:1261] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 1 FOLLOWER]: Refusing update from remote peer 79ed53dd5b6f4276b9e248464cbe6a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:21.163630 14398 consensus_queue.cc:1035] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe71c4b19e648b8b55f34e7d7dce36f" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:21.163774 14400 consensus_queue.cc:1035] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "780aa75b3bcc4224b433075d43ef2743" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:21.166779 14405 mvcc.cc:205] Tried to move back new op lower bound from 7333329310362271744 to 7333329310287302656. Current Snapshot: MvccSnapshot[applied={T|T < 7333329310362271744 or (T in {7333329310362271744})}]
I20260925 19:13:21.168680 13916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:21.168773 13916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:21.169458 13916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46149a418bb84977b6df3f8ce3823edd: Sending DeleteTablet for 3 replicas of tablet aeb1bf2be5e04bc3873a54fc76642d88
I20260925 19:13:21.169817 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57478
I20260925 19:13:21.169821 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:48866
I20260925 19:13:21.170049 14176 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb1bf2be5e04bc3873a54fc76642d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:41320
I20260925 19:13:21.170064 14413 tablet_replica.cc:330] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: stopping tablet replica
I20260925 19:13:21.170181 14413 raft_consensus.cc:2229] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:13:21.170302 14413 raft_consensus.cc:2258] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:21.170300 14415 tablet_replica.cc:330] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: stopping tablet replica
I20260925 19:13:21.170440 14414 tablet_replica.cc:330] T aeb1bf2be5e04bc3873a54fc76642d88 P afe71c4b19e648b8b55f34e7d7dce36f: stopping tablet replica
I20260925 19:13:21.170490 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13979
I20260925 19:13:21.170442 14415 raft_consensus.cc:2229] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:21.171507 14415 raft_consensus.cc:2258] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:21.171991 14415 ts_tablet_manager.cc:1905] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:21.173693 14413 ts_tablet_manager.cc:1905] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:21.174185 14415 ts_tablet_manager.cc:1918] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:21.174306 14415 log.cc:1196] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/aeb1bf2be5e04bc3873a54fc76642d88
I20260925 19:13:21.174628 14415 ts_tablet_manager.cc:1939] T aeb1bf2be5e04bc3873a54fc76642d88 P 780aa75b3bcc4224b433075d43ef2743: Deleting consensus metadata
I20260925 19:13:21.175184 13904 catalog_manager.cc:4841] TS 780aa75b3bcc4224b433075d43ef2743 (127.27.174.66:37643): tablet aeb1bf2be5e04bc3873a54fc76642d88 (table test-table [id=86e98958d3d74b4f9a27529c7e94b7c5]) successfully deleted
I20260925 19:13:21.175288 14413 ts_tablet_manager.cc:1918] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:21.175364 14413 log.cc:1196] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/aeb1bf2be5e04bc3873a54fc76642d88
I20260925 19:13:21.175658 14413 ts_tablet_manager.cc:1939] T aeb1bf2be5e04bc3873a54fc76642d88 P 79ed53dd5b6f4276b9e248464cbe6a0f: Deleting consensus metadata
I20260925 19:13:21.176092 13902 catalog_manager.cc:4841] TS 79ed53dd5b6f4276b9e248464cbe6a0f (127.27.174.67:46253): tablet aeb1bf2be5e04bc3873a54fc76642d88 (table test-table [id=86e98958d3d74b4f9a27529c7e94b7c5]) successfully deleted
W20260925 19:13:21.177139 13901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:38275 (error 108)
I20260925 19:13:21.177345 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14113
I20260925 19:13:21.180339 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.180222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38275 (local address 127.0.0.1:48870)
0925 19:13:21.180272 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.180313 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:13:21.180512 13901 catalog_manager.cc:4584] TS afe71c4b19e648b8b55f34e7d7dce36f (127.27.174.65:38275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aeb1bf2be5e04bc3873a54fc76642d88: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38275: connect: Connection refused (error 111)
I20260925 19:13:21.183734 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14247
I20260925 19:13:21.188184 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 13885
2026-09-25T19:13:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1930 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T19:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:21Z Disabled control of system clock
I20260925 19:13:21.214394 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35233
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:21.293148 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:21.293360 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:21.293381 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:21.294659 14420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:21.294700 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:21.294713 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:21.294726 14420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:21.294772 14420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:21.296090 14420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:21.296340 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:21.296622 14420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:21.301312 14426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:21.301399 14428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.301469 14420 server_base.cc:1034] running on GCE node
W20260925 19:13:21.301319 14425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.301882 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:21.302167 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:21.303426 14420 hybrid_clock.cc:648] HybridClock initialized: now 1790363601303406 us; error 35 us; skew 500 ppm
I20260925 19:13:21.311210 14420 webserver.cc:460] Webserver started at http://127.27.174.126:41371/ using document root <none> and password file <none>
I20260925 19:13:21.311463 14420 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:21.311512 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:21.311705 14420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:21.312781 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "cf82e92852124fe0af8a4f6c8e3f0de6"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "47c3f09a4ba427d6a82b68c033a72501"
server_key_iv: "6aa2b57f59826da06e3db9a0a39d333b"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.313134 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "cf82e92852124fe0af8a4f6c8e3f0de6"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "47c3f09a4ba427d6a82b68c033a72501"
server_key_iv: "6aa2b57f59826da06e3db9a0a39d333b"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.314368 14420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:21.315062 14434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.315255 14420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:21.315351 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "cf82e92852124fe0af8a4f6c8e3f0de6"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "47c3f09a4ba427d6a82b68c033a72501"
server_key_iv: "6aa2b57f59826da06e3db9a0a39d333b"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.315425 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:21.327816 14420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:21.328087 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:21.328243 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:21.332010 14420 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35233
I20260925 19:13:21.332043 14486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35233 every 8 connection(s)
I20260925 19:13:21.332356 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:21.333257 14487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.335371 14487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: Bootstrap starting.
I20260925 19:13:21.335911 14487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.336151 14487 log.cc:824] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.336752 14487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: No bootstrap required, opened a new log
I20260925 19:13:21.338255 14487 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } }
I20260925 19:13:21.338462 14487 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.338488 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14420
I20260925 19:13:21.338510 14487 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf82e92852124fe0af8a4f6c8e3f0de6, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.338590 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:21.338666 14487 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } }
I20260925 19:13:21.338749 14487 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:21.338780 28345 external_mini_cluster.cc:1428] Setting key 6de9dab0618e0dfc820142ea198d0f2b
I20260925 19:13:21.338779 14487 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:21.338835 14487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.339515 14487 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } }
I20260925 19:13:21.339670 14487 leader_election.cc:302] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf82e92852124fe0af8a4f6c8e3f0de6; no voters:
I20260925 19:13:21.339841 14487 leader_election.cc:288] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:21.340005 14493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:21.340112 14487 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:21.340143 14493 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [term 1 LEADER]: Becoming Leader. State: Replica: cf82e92852124fe0af8a4f6c8e3f0de6, State: Running, Role: LEADER
I20260925 19:13:21.340224 14493 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } }
I20260925 19:13:21.340662 14492 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } } }
I20260925 19:13:21.340749 14492 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:21.340965 14501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:21.340958 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf82e92852124fe0af8a4f6c8e3f0de6. Latest consensus state: current_term: 1 leader_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf82e92852124fe0af8a4f6c8e3f0de6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35233 } } }
I20260925 19:13:21.341298 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:21.341476 14501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:21.343698 14501 catalog_manager.cc:1347] Generated new cluster ID: 09b0f7b4cf6b4c949ffbce9d3d6e2fa2
I20260925 19:13:21.343753 14501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:21.344887 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.339264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.27.174.126:35233)
0925 19:13:21.339473 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.339480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.339503 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:13:21.339589 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.339594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.339715 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.339857 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.340108 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.341810 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.342924 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.343384 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.343522 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.344229 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.344233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.344233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:21.344403 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.344434 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:21.344444 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.344510 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.344591 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.344674 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260925 19:13:21.374939 14501 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:21.375473 14501 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:21.382740 14501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: Generated new TSK 0
I20260925 19:13:21.382925 14501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:21.384341 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--builtin_ntp_servers=127.27.174.84:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:21.462842 14511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:21.463059 14511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:21.463093 14511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:21.464345 14511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:21.464396 14511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:21.464437 14511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:21.465747 14511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:21.465996 14511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:21.466279 14511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:21.471802 14517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:21.471801 14519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:21.471877 14516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.472146 14511 server_base.cc:1034] running on GCE node
I20260925 19:13:21.472375 14511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:21.472591 14511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:21.473744 14511 hybrid_clock.cc:648] HybridClock initialized: now 1790363601473731 us; error 30 us; skew 500 ppm
I20260925 19:13:21.474995 14511 webserver.cc:460] Webserver started at http://127.27.174.65:46171/ using document root <none> and password file <none>
I20260925 19:13:21.475225 14511 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:21.475289 14511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:21.475435 14511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:21.476495 14511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "f6107fd26feb47b695e94e01b443e937"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "7613f012c2940f843518e831d715d4b1"
server_key_iv: "b044676b7d5c3518652131eebf45d793"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.476845 14511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "f6107fd26feb47b695e94e01b443e937"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "7613f012c2940f843518e831d715d4b1"
server_key_iv: "b044676b7d5c3518652131eebf45d793"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.478082 14511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:13:21.478803 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.478950 14511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:21.479031 14511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "f6107fd26feb47b695e94e01b443e937"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "7613f012c2940f843518e831d715d4b1"
server_key_iv: "b044676b7d5c3518652131eebf45d793"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.479084 14511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:21.500389 14511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:21.500659 14511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:21.500763 14511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:21.500959 14511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:21.501248 14511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:21.501278 14511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.501300 14511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:21.501312 14511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.506981 14511 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:45013
I20260925 19:13:21.507023 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:45013 every 8 connection(s)
I20260925 19:13:21.507362 14511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:21.509251 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14511
I20260925 19:13:21.509361 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:21.509542 28345 external_mini_cluster.cc:1428] Setting key 5c39da38e8be25ae1f32c21bfd3ffe9b
I20260925 19:13:21.510429 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--builtin_ntp_servers=127.27.174.84:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:13:21.513082 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.508083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:59363 (local address 127.27.174.126:35233)
0925 19:13:21.508149 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.508150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.508381 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:13:21.508928 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.508930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.508945 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.508993 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.509771 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.510367 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.511402 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.511705 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.511771 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.512796 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.512798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.512798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:21.512805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.512822 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:21.512826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.512888 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.512952 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.512983 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:13:21.513274 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.508080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35233 (local address 127.27.174.65:59363)
0925 19:13:21.508262 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.508298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.508823 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.509080 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.509084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.509099 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:21.509673 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.509679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.510454 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.510455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.511317 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.511319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.511792 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.511793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.511933 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.512552 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:21.512553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:21.512588 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:21.512612 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:21.512894 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:21.512899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:21.512901 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.512953 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.513039 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260925 19:13:21.513710 14639 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35233
I20260925 19:13:21.513796 14639 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:21.513993 14639 heartbeater.cc:511] Master 127.27.174.126:35233 requested a full tablet report, sending...
I20260925 19:13:21.514526 14451 ts_manager.cc:194] Registered new tserver with Master: f6107fd26feb47b695e94e01b443e937 (127.27.174.65:45013)
I20260925 19:13:21.515144 14451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:59363
W20260925 19:13:21.590557 14642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:21.590737 14642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:21.590757 14642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:21.591905 14642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:21.591941 14642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:21.591986 14642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:21.593283 14642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:21.593505 14642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:21.593780 14642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:21.598392 14650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:21.598419 14648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.598578 14642 server_base.cc:1034] running on GCE node
W20260925 19:13:21.598618 14647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.598867 14642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:21.599050 14642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:21.600207 14642 hybrid_clock.cc:648] HybridClock initialized: now 1790363601600199 us; error 32 us; skew 500 ppm
I20260925 19:13:21.601473 14642 webserver.cc:460] Webserver started at http://127.27.174.66:43307/ using document root <none> and password file <none>
I20260925 19:13:21.601715 14642 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:21.601784 14642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:21.601922 14642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:21.602918 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "2a4873a749c0491cb5ff0e4993663f3f"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "4d0e48763c13016f8d211075c52bd431"
server_key_iv: "f3234fdfd6cfd94724d40f40a5e3b640"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.603282 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4873a749c0491cb5ff0e4993663f3f"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "4d0e48763c13016f8d211075c52bd431"
server_key_iv: "f3234fdfd6cfd94724d40f40a5e3b640"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.604615 14642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:21.605363 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.605530 14642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:21.605628 14642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "2a4873a749c0491cb5ff0e4993663f3f"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "4d0e48763c13016f8d211075c52bd431"
server_key_iv: "f3234fdfd6cfd94724d40f40a5e3b640"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.605695 14642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:21.619704 14642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:21.619958 14642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:21.620103 14642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:21.620306 14642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:21.620602 14642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:21.620653 14642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.620687 14642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:21.620713 14642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.626125 14642 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:33069
I20260925 19:13:21.626215 14769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:33069 every 8 connection(s)
I20260925 19:13:21.626497 14642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:13:21.631101 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.627178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37461 (local address 127.27.174.126:35233)
0925 19:13:21.627237 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.627239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.627453 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:13:21.627926 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.627927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.627936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.627992 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.628603 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.629080 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.629989 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.630168 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.630218 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.630884 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.630886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.630886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:21.630890 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.630902 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:21.630906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.630949 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.631023 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.631044 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:13:21.631311 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.627152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35233 (local address 127.27.174.66:37461)
0925 19:13:21.627333 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.627372 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.627843 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.628041 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.628044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.628052 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:21.628519 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.628523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.629155 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.629156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.629909 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.629911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.630235 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.630236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.630335 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.630767 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:21.630767 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:21.630793 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:21.630813 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:21.630957 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:21.630961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:21.630963 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.631012 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.631083 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260925 19:13:21.631798 14770 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35233
I20260925 19:13:21.631870 14770 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:21.632047 14770 heartbeater.cc:511] Master 127.27.174.126:35233 requested a full tablet report, sending...
I20260925 19:13:21.632395 14451 ts_manager.cc:194] Registered new tserver with Master: 2a4873a749c0491cb5ff0e4993663f3f (127.27.174.66:33069)
I20260925 19:13:21.632755 14451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:37461
I20260925 19:13:21.635538 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14642
I20260925 19:13:21.635677 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:21.635867 28345 external_mini_cluster.cc:1428] Setting key 6724625c16392b45a70b3a5fef01fe1b
I20260925 19:13:21.636675 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--builtin_ntp_servers=127.27.174.84:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:21.716631 14773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:21.716825 14773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:21.716845 14773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:21.718079 14773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:21.718119 14773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:21.718145 14773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:21.719832 14773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:21.720059 14773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:21.720337 14773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:21.725230 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.725260 14773 server_base.cc:1034] running on GCE node
W20260925 19:13:21.725225 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:21.725242 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:21.725579 14773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:21.725809 14773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:21.726962 14773 hybrid_clock.cc:648] HybridClock initialized: now 1790363601726947 us; error 28 us; skew 500 ppm
I20260925 19:13:21.728175 14773 webserver.cc:460] Webserver started at http://127.27.174.67:44773/ using document root <none> and password file <none>
I20260925 19:13:21.728394 14773 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:21.728457 14773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:21.728600 14773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:21.729629 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "5a9e06491cd34f6f9efcf308fbc1069a"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "dbad2b52d02b47bd6b258cd4b6ed4c71"
server_key_iv: "7e03f4927cadce491709ac530aa903f8"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.729993 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "5a9e06491cd34f6f9efcf308fbc1069a"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "dbad2b52d02b47bd6b258cd4b6ed4c71"
server_key_iv: "7e03f4927cadce491709ac530aa903f8"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.731221 14773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:13:21.731917 14787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.732071 14773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.732165 14773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "5a9e06491cd34f6f9efcf308fbc1069a"
format_stamp: "Formatted at 2026-09-25 19:13:21 on dist-test-slave-fcdn"
server_key: "dbad2b52d02b47bd6b258cd4b6ed4c71"
server_key_iv: "7e03f4927cadce491709ac530aa903f8"
server_key_version: "encryptionkey@0"
I20260925 19:13:21.732231 14773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:21.747766 14773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:21.748034 14773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:21.748167 14773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:21.748365 14773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:21.748679 14773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:21.748730 14773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.748769 14773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:21.748797 14773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:21.754315 14773 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:34909
I20260925 19:13:21.754355 14900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:34909 every 8 connection(s)
I20260925 19:13:21.754669 14773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:21.759470 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.755337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:36827 (local address 127.27.174.126:35233)
0925 19:13:21.755390 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.755393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.755630 (+ 237us) server_negotiation.cc:407] Connection header received
0925 19:13:21.756126 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.756128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.756138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.756190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.756801 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.757286 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.758320 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.758505 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.758561 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.759253 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.759254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.759254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:21.759258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.759271 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:21.759275 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.759318 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.759392 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.759414 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:21.759686 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.755328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35233 (local address 127.27.174.67:36827)
0925 19:13:21.755501 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.755533 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.756040 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.756241 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.756244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.756252 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:21.756731 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.756735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.757358 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.757359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.758231 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.758234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.758574 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.758574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.758684 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.759132 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:21.759133 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:21.759158 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:21.759179 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:21.759326 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:21.759330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:21.759332 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.759382 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.759455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260925 19:13:21.760105 14901 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35233
I20260925 19:13:21.760203 14901 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:21.760368 14901 heartbeater.cc:511] Master 127.27.174.126:35233 requested a full tablet report, sending...
I20260925 19:13:21.760722 14451 ts_manager.cc:194] Registered new tserver with Master: 5a9e06491cd34f6f9efcf308fbc1069a (127.27.174.67:34909)
I20260925 19:13:21.760988 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14773
I20260925 19:13:21.761107 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:21.761101 14451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:36827
I20260925 19:13:21.761296 28345 external_mini_cluster.cc:1428] Setting key f1870178fa016d97410fa6fe9cc7665b
I20260925 19:13:21.762156 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:21.766179 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.763445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.27.174.126:35233)
0925 19:13:21.763506 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.763508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.763630 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:13:21.763706 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.763707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.763716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.763758 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:21.763874 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.764310 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.765129 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.765305 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.765356 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.765968 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:21.765969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:21.765969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:21.765973 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:21.765983 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:21.765986 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:21.766045 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.766116 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.766124 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:13:21.767738 14451 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:21.769161 14451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:21.774967 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.27.174.67:34909)
0925 19:13:21.771500 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.771504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.771512 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:21.771537 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.771538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.771549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.771664 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.771836 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.772914 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.774001 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.774579 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.774650 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.774726 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.774858 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.774892 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":77,"threads_started":1}
I20260925 19:13:21.774958 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45013 (local address 127.0.0.1:39342)
0925 19:13:21.771349 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.771358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.771397 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.771772 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.771773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.771775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.771818 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.771820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.772755 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.772756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.773810 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.773812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.774733 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.774734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.774782 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.774784 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.774812 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.774863 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":76,"threads_started":1}
I20260925 19:13:21.774958 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:34909 (local address 127.0.0.1:52004)
0925 19:13:21.771239 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.771277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.771343 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.771671 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.771674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.771676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.771743 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.771745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.773006 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.773007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.773855 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.773859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.774668 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.774670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.774753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.774766 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.774838 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.774867 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":54,"threads_started":1}
I20260925 19:13:21.775390 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:33069 (local address 127.0.0.1:51216)
0925 19:13:21.771742 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.771752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.771798 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.772062 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.772063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.772065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.772099 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.772100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.773281 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.773282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.774302 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.774304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.775183 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.775184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.775244 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.775248 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.775288 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.775324 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":66,"threads_started":1}
I20260925 19:13:21.775403 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.27.174.66:33069)
0925 19:13:21.771537 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.771542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.771862 (+ 320us) server_negotiation.cc:407] Connection header received
0925 19:13:21.771915 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.771917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.771930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.771998 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.772171 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.773198 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.774384 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.775113 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.775180 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.775260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.775295 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.775327 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260925 19:13:21.775619 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.771172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.27.174.65:45013)
0925 19:13:21.771581 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.771585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.771594 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:21.771661 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.771663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.771678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.771773 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.771901 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.772670 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.773958 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.774615 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.775362 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.775459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.775484 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.775522 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":88,"threads_started":1}
I20260925 19:13:21.775859 14835 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 (DEFAULT_TABLE table=test-table [id=17e5399ca64d43b7a8ced3c4bde60ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.776041 14704 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 (DEFAULT_TABLE table=test-table [id=17e5399ca64d43b7a8ced3c4bde60ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.776229 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0d5f30d68749cead26ae1b0d000231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.776225 14573 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 (DEFAULT_TABLE table=test-table [id=17e5399ca64d43b7a8ced3c4bde60ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:21.776396 14704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0d5f30d68749cead26ae1b0d000231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.776535 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0d5f30d68749cead26ae1b0d000231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:21.779122 14915 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: Bootstrap starting.
I20260925 19:13:21.779417 14916 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Bootstrap starting.
I20260925 19:13:21.779842 14915 tablet_bootstrap.cc:654] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.780054 14916 tablet_bootstrap.cc:654] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.780189 14915 log.cc:824] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.780360 14916 log.cc:824] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.780679 14917 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Bootstrap starting.
I20260925 19:13:21.781004 14915 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: No bootstrap required, opened a new log
I20260925 19:13:21.781023 14916 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: No bootstrap required, opened a new log
I20260925 19:13:21.781090 14916 ts_tablet_manager.cc:1397] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:21.781086 14915 ts_tablet_manager.cc:1397] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:21.781352 14917 tablet_bootstrap.cc:654] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:21.781711 14917 log.cc:824] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:21.782495 14917 tablet_bootstrap.cc:492] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: No bootstrap required, opened a new log
I20260925 19:13:21.782572 14917 ts_tablet_manager.cc:1397] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:21.783384 14916 raft_consensus.cc:348] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.783527 14916 raft_consensus.cc:374] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.783566 14916 raft_consensus.cc:729] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9e06491cd34f6f9efcf308fbc1069a, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.783710 14916 consensus_queue.cc:260] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.783957 14916 ts_tablet_manager.cc:1428] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:21.783872 14915 raft_consensus.cc:348] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.784017 14915 raft_consensus.cc:374] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.784070 14915 raft_consensus.cc:729] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6107fd26feb47b695e94e01b443e937, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.784078 14901 heartbeater.cc:503] Master 127.27.174.126:35233 was elected leader, sending a full tablet report...
I20260925 19:13:21.784176 14915 consensus_queue.cc:260] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.784413 14915 ts_tablet_manager.cc:1428] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:21.784453 14639 heartbeater.cc:503] Master 127.27.174.126:35233 was elected leader, sending a full tablet report...
I20260925 19:13:21.785115 14917 raft_consensus.cc:348] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.785264 14917 raft_consensus.cc:374] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:21.785305 14917 raft_consensus.cc:729] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4873a749c0491cb5ff0e4993663f3f, State: Initialized, Role: FOLLOWER
I20260925 19:13:21.785391 14917 consensus_queue.cc:260] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.785573 14917 ts_tablet_manager.cc:1428] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:21.785622 14770 heartbeater.cc:503] Master 127.27.174.126:35233 was elected leader, sending a full tablet report...
W20260925 19:13:21.877012 14771 tablet.cc:2369] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:21.905932 14923 raft_consensus.cc:482] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:21.906052 14923 raft_consensus.cc:504] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.906378 14923 leader_election.cc:288] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9e06491cd34f6f9efcf308fbc1069a (127.27.174.67:34909), f6107fd26feb47b695e94e01b443e937 (127.27.174.65:45013)
I20260925 19:13:21.910245 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.906634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:42567 (local address 127.27.174.65:45013)
0925 19:13:21.906689 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.906691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.906699 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:21.906735 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.906736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.906745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.906804 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.906984 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.907795 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909333 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.909882 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909975 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.909983 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.910135 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.910177 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:13:21.910245 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.906638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44525 (local address 127.27.174.67:34909)
0925 19:13:21.906681 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.906683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.906883 (+ 200us) server_negotiation.cc:407] Connection header received
0925 19:13:21.906910 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.906910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.906917 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.906976 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:21.907100 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.907807 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909333 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.909887 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909975 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.909983 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.910135 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.910176 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:13:21.910279 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.906563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45013 (local address 127.27.174.66:42567)
0925 19:13:21.906610 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.906623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.906666 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.906832 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.906833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.906835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.906909 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.906915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.907871 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.907871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.909233 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.909236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.909981 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.910059 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.910065 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.910120 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.910172 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":71}
I20260925 19:13:21.910290 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.906571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:34909 (local address 127.27.174.66:44525)
0925 19:13:21.906778 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.906803 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:21.906854 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:21.906979 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:21.906980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:21.906982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:21.907040 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.907041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.907871 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.907871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.909233 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:21.909236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.909982 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.909983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:21.910059 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.910065 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:13:21.910120 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:13:21.910167 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":103,"thread_start_us":104,"threads_started":1}
I20260925 19:13:21.910706 14855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0d5f30d68749cead26ae1b0d000231" candidate_uuid: "2a4873a749c0491cb5ff0e4993663f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" is_pre_election: true
I20260925 19:13:21.910857 14855 raft_consensus.cc:2454] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4873a749c0491cb5ff0e4993663f3f in term 0.
I20260925 19:13:21.910959 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0d5f30d68749cead26ae1b0d000231" candidate_uuid: "2a4873a749c0491cb5ff0e4993663f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6107fd26feb47b695e94e01b443e937" is_pre_election: true
I20260925 19:13:21.911051 14658 leader_election.cc:302] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4873a749c0491cb5ff0e4993663f3f, 5a9e06491cd34f6f9efcf308fbc1069a; no voters:
I20260925 19:13:21.911095 14593 raft_consensus.cc:2454] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a4873a749c0491cb5ff0e4993663f3f in term 0.
I20260925 19:13:21.911157 14923 raft_consensus.cc:2781] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:21.911201 14923 raft_consensus.cc:482] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:21.911226 14923 raft_consensus.cc:3037] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.911974 14923 raft_consensus.cc:504] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.912114 14923 leader_election.cc:288] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [CANDIDATE]: Term 1 election: Requested vote from peers 5a9e06491cd34f6f9efcf308fbc1069a (127.27.174.67:34909), f6107fd26feb47b695e94e01b443e937 (127.27.174.65:45013)
I20260925 19:13:21.912263 14855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0d5f30d68749cead26ae1b0d000231" candidate_uuid: "2a4873a749c0491cb5ff0e4993663f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9e06491cd34f6f9efcf308fbc1069a"
I20260925 19:13:21.912277 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0d5f30d68749cead26ae1b0d000231" candidate_uuid: "2a4873a749c0491cb5ff0e4993663f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6107fd26feb47b695e94e01b443e937"
I20260925 19:13:21.912340 14855 raft_consensus.cc:3037] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.912344 14593 raft_consensus.cc:3037] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:21.913018 14855 raft_consensus.cc:2454] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4873a749c0491cb5ff0e4993663f3f in term 1.
I20260925 19:13:21.913125 14593 raft_consensus.cc:2454] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a4873a749c0491cb5ff0e4993663f3f in term 1.
I20260925 19:13:21.913231 14658 leader_election.cc:302] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4873a749c0491cb5ff0e4993663f3f, 5a9e06491cd34f6f9efcf308fbc1069a; no voters:
I20260925 19:13:21.913333 14923 raft_consensus.cc:2781] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:21.913479 14923 raft_consensus.cc:686] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 1 LEADER]: Becoming Leader. State: Replica: 2a4873a749c0491cb5ff0e4993663f3f, State: Running, Role: LEADER
I20260925 19:13:21.913585 14923 consensus_queue.cc:237] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } }
I20260925 19:13:21.914354 14450 catalog_manager.cc:5495] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a4873a749c0491cb5ff0e4993663f3f (127.27.174.66). New cstate: current_term: 1 leader_uuid: "2a4873a749c0491cb5ff0e4993663f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a4873a749c0491cb5ff0e4993663f3f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 33069 } health_report { overall_health: HEALTHY } } }
I20260925 19:13:21.954309 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.952188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.27.174.66:33069)
0925 19:13:21.952236 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:21.952238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:21.952246 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:21.952305 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:21.952305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:21.952323 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:21.952383 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:21.952495 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.952950 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.953657 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:21.953839 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:21.953892 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:21.954009 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:21.954090 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:21.954161 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:21.954215 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:13:21.954223 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:13:21.955955 14855 raft_consensus.cc:1261] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 1 FOLLOWER]: Refusing update from remote peer 2a4873a749c0491cb5ff0e4993663f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:21.955979 14593 raft_consensus.cc:1261] T 9c0d5f30d68749cead26ae1b0d000231 P f6107fd26feb47b695e94e01b443e937 [term 1 FOLLOWER]: Refusing update from remote peer 2a4873a749c0491cb5ff0e4993663f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:21.956302 14925 consensus_queue.cc:1035] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6107fd26feb47b695e94e01b443e937" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:21.956426 14923 consensus_queue.cc:1035] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9e06491cd34f6f9efcf308fbc1069a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:21.958587 14928 mvcc.cc:205] Tried to move back new op lower bound from 7333329313609203712 to 7333329313438724096. Current Snapshot: MvccSnapshot[applied={T|T < 7333329313609203712}]
I20260925 19:13:21.958971 14930 mvcc.cc:205] Tried to move back new op lower bound from 7333329313609203712 to 7333329313438724096. Current Snapshot: MvccSnapshot[applied={T|T < 7333329313609203712}]
I20260925 19:13:21.959043 14929 mvcc.cc:205] Tried to move back new op lower bound from 7333329313609203712 to 7333329313438724096. Current Snapshot: MvccSnapshot[applied={T|T < 7333329313609203712}]
I20260925 19:13:21.961650 14450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:21.961721 14450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:21.962402 14450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf82e92852124fe0af8a4f6c8e3f0de6: Sending DeleteTablet for 3 replicas of tablet 9c0d5f30d68749cead26ae1b0d000231
I20260925 19:13:21.962759 14704 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='slave'} at 127.0.0.1:51216
I20260925 19:13:21.962878 14835 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='slave'} at 127.0.0.1:52004
I20260925 19:13:21.963009 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14511
I20260925 19:13:21.963088 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0d5f30d68749cead26ae1b0d000231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:21 UTC) from {username='slave'} at 127.0.0.1:39342
I20260925 19:13:21.963122 14939 tablet_replica.cc:330] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: stopping tablet replica
I20260925 19:13:21.963274 14939 raft_consensus.cc:2229] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:21.963572 14938 tablet_replica.cc:330] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: stopping tablet replica
I20260925 19:13:21.963683 14938 raft_consensus.cc:2229] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:13:21.963796 14938 raft_consensus.cc:2258] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:21.964392 14938 ts_tablet_manager.cc:1905] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:21.964620 14939 raft_consensus.cc:2258] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:21.965258 14939 ts_tablet_manager.cc:1905] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:21.966063 14938 ts_tablet_manager.cc:1918] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:21.966121 14938 log.cc:1196] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/9c0d5f30d68749cead26ae1b0d000231
I20260925 19:13:21.966346 14938 ts_tablet_manager.cc:1939] T 9c0d5f30d68749cead26ae1b0d000231 P 2a4873a749c0491cb5ff0e4993663f3f: Deleting consensus metadata
I20260925 19:13:21.966643 14436 catalog_manager.cc:4841] TS 2a4873a749c0491cb5ff0e4993663f3f (127.27.174.66:33069): tablet 9c0d5f30d68749cead26ae1b0d000231 (table test-table [id=17e5399ca64d43b7a8ced3c4bde60ec4]) successfully deleted
I20260925 19:13:21.967621 14939 ts_tablet_manager.cc:1918] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:21.967698 14939 log.cc:1196] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/9c0d5f30d68749cead26ae1b0d000231
I20260925 19:13:21.967932 14939 ts_tablet_manager.cc:1939] T 9c0d5f30d68749cead26ae1b0d000231 P 5a9e06491cd34f6f9efcf308fbc1069a: Deleting consensus metadata
I20260925 19:13:21.968315 14436 catalog_manager.cc:4841] TS 5a9e06491cd34f6f9efcf308fbc1069a (127.27.174.67:34909): tablet 9c0d5f30d68749cead26ae1b0d000231 (table test-table [id=17e5399ca64d43b7a8ced3c4bde60ec4]) successfully deleted
W20260925 19:13:21.969383 14435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:45013 (error 108)
I20260925 19:13:21.969620 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14642
I20260925 19:13:21.969821 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:21.969712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45013 (local address 127.0.0.1:39348)
0925 19:13:21.969756 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:21.969790 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:13:21.970520 14435 catalog_manager.cc:4584] TS f6107fd26feb47b695e94e01b443e937 (127.27.174.65:45013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c0d5f30d68749cead26ae1b0d000231: Network error: Client connection negotiation failed: client connection to 127.27.174.65:45013: connect: Connection refused (error 111)
I20260925 19:13:21.974567 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14773
I20260925 19:13:21.978685 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14420
2026-09-25T19:13:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (789 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:22 dist-test-slave-fcdn krb5kdc[14943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:22 dist-test-slave-fcdn krb5kdc[14943](info): set up 2 sockets
Sep 25 19:13:22 dist-test-slave-fcdn krb5kdc[14943](info): commencing operation
krb5kdc: starting...
W20260925 19:13:23.007417 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:23.122562 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45275
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:23.201915 14956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:23.202114 14956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:23.202137 14956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:23.203269 14956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:23.203310 14956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:23.203325 14956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:23.203337 14956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:23.203387 14956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:23.204838 14956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:23.205082 14956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:23.205299 14956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:23.209973 14961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.209991 14964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.209991 14962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:23.210219 14956 server_base.cc:1034] running on GCE node
I20260925 19:13:23.210456 14956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:23.210723 14956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:23.211894 14956 hybrid_clock.cc:648] HybridClock initialized: now 1790363603211864 us; error 53 us; skew 500 ppm
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:23.214345 14956 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:23.214808 14956 webserver.cc:460] Webserver started at http://127.27.174.126:39671/ using document root <none> and password file <none>
I20260925 19:13:23.215058 14956 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:23.215123 14956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:23.215253 14956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:23.216320 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "b122816faad641ce8a7290ab29e60cfa"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "69979ad6dc2afa5e2589ddff3435fdf3"
server_key_iv: "afa5fb0ab254e41755c52ee488ab1b3e"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.216715 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "b122816faad641ce8a7290ab29e60cfa"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "69979ad6dc2afa5e2589ddff3435fdf3"
server_key_iv: "afa5fb0ab254e41755c52ee488ab1b3e"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.218003 14956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:23.218657 14971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.218822 14956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:23.218915 14956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "b122816faad641ce8a7290ab29e60cfa"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "69979ad6dc2afa5e2589ddff3435fdf3"
server_key_iv: "afa5fb0ab254e41755c52ee488ab1b3e"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.219031 14956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:23.228595 14956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:23.230657 14956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:23.230799 14956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:23.234571 14956 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45275
I20260925 19:13:23.234601 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45275 every 8 connection(s)
I20260925 19:13:23.234908 14956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:23.235737 15024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:23.236780 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 14956
I20260925 19:13:23.236886 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:23.237076 28345 external_mini_cluster.cc:1428] Setting key 43bdb0fcf600d0740fa3f7d51e1fd7d9
I20260925 19:13:23.238204 15024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: Bootstrap starting.
I20260925 19:13:23.238914 15024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:23.239238 15024 log.cc:824] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:23.240000 15024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: No bootstrap required, opened a new log
I20260925 19:13:23.242259 15024 raft_consensus.cc:348] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } }
I20260925 19:13:23.242444 15024 raft_consensus.cc:374] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:23.242489 15024 raft_consensus.cc:729] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b122816faad641ce8a7290ab29e60cfa, State: Initialized, Role: FOLLOWER
I20260925 19:13:23.242625 15024 consensus_queue.cc:260] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } }
I20260925 19:13:23.242715 15024 raft_consensus.cc:388] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:23.242755 15024 raft_consensus.cc:482] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:23.242811 15024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:23.243506 15024 raft_consensus.cc:504] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } }
I20260925 19:13:23.243592 15024 leader_election.cc:302] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b122816faad641ce8a7290ab29e60cfa; no voters:
I20260925 19:13:23.243783 15024 leader_election.cc:288] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:23.243813 15029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:23.243925 15029 raft_consensus.cc:686] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [term 1 LEADER]: Becoming Leader. State: Replica: b122816faad641ce8a7290ab29e60cfa, State: Running, Role: LEADER
I20260925 19:13:23.244000 15029 consensus_queue.cc:237] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } }
I20260925 19:13:23.244027 15024 sys_catalog.cc:564] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:23.244292 15029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b122816faad641ce8a7290ab29e60cfa. Latest consensus state: current_term: 1 leader_uuid: "b122816faad641ce8a7290ab29e60cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } } }
I20260925 19:13:23.244360 15029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:23.244309 15031 sys_catalog.cc:455] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b122816faad641ce8a7290ab29e60cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b122816faad641ce8a7290ab29e60cfa" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45275 } } }
I20260925 19:13:23.244525 15031 sys_catalog.cc:458] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:23.245249 15033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:23.245819 15033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:23.247617 15033 catalog_manager.cc:1347] Generated new cluster ID: 5c510983905b4b6e8856c14f1b152d77
I20260925 19:13:23.247670 15033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:23.248337 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.237549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.27.174.126:45275)
0925 19:13:23.237716 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.237723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.237745 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:13:23.238167 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.238173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.238216 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.238356 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:23.239167 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.240372 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.241409 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.241788 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.241863 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.244665 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:23.244670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:23.244670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:23.244682 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:23.246560 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.246951 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.246953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.246953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.247043 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.247257 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.247258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.247259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.247470 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:23.248036 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.248102 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.248202 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260925 19:13:23.270347 15033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:23.271009 15033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:23.281373 15033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: Generated new TSK 0
I20260925 19:13:23.281571 15033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20260925 19:13:23.314999 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:23.420605 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:23.420794 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:23.420815 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:23.422016 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:23.422055 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:23.422103 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:23.423558 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:23.423833 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:23.424052 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:23.428912 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.428915 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.428920 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:23.429122 15050 server_base.cc:1034] running on GCE node
I20260925 19:13:23.429332 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:23.429565 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:23.430733 15050 hybrid_clock.cc:648] HybridClock initialized: now 1790363603430724 us; error 39 us; skew 500 ppm
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:23.433176 15050 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20260925 19:13:23.433612 15050 webserver.cc:460] Webserver started at http://127.27.174.65:46335/ using document root <none> and password file <none>
I20260925 19:13:23.433827 15050 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:23.433889 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:23.434023 15050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:23.435039 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "f5d6506093a8474bb9f5bb911cc7e913"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "b19b31bc082bd8b168a0912d08d78eef"
server_key_iv: "3150a54039e2b20e6782aab534a50cad"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.435391 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d6506093a8474bb9f5bb911cc7e913"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "b19b31bc082bd8b168a0912d08d78eef"
server_key_iv: "3150a54039e2b20e6782aab534a50cad"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.436726 15050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:23.437422 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.437599 15050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:23.437691 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "f5d6506093a8474bb9f5bb911cc7e913"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "b19b31bc082bd8b168a0912d08d78eef"
server_key_iv: "3150a54039e2b20e6782aab534a50cad"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.437772 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:23.459899 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:23.461752 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:23.461910 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:23.462142 15050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:23.462435 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:23.462484 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.462536 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:23.462563 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.468152 15050 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:43231
I20260925 19:13:23.468195 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:43231 every 8 connection(s)
I20260925 19:13:23.468571 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:23.470347 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15050
I20260925 19:13:23.470458 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:23.470657 28345 external_mini_cluster.cc:1428] Setting key 9bb11b962201f29b428abb0722fda4c5
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:23.478758 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.469336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:55145 (local address 127.27.174.126:45275)
0925 19:13:23.469409 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.469412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.469605 (+ 193us) server_negotiation.cc:407] Connection header received
0925 19:13:23.470152 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.470153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.470166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.470238 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:23.471474 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.472036 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.473025 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.473253 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.473326 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.475791 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:23.475793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:23.475793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:23.475801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:23.477370 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.477750 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.477752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.477752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.477817 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.478079 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.478081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.478081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.478223 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:23.478294 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.478552 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.478642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:13:23.478772 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.469308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45275 (local address 127.27.174.65:55145)
0925 19:13:23.469471 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.469524 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.470046 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.470320 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.470328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.471111 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:23.471388 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.471394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.472126 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.472127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.472930 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.472932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.473334 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.473335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.473481 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.473992 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:23.474002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:23.475703 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:23.477450 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.477456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.477458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.477677 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.477900 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.477901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.477901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.477997 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.478297 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:23.478301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:23.478413 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.478491 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.478613 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260925 19:13:23.479271 15179 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45275
I20260925 19:13:23.479373 15179 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20260925 19:13:23.479619 15179 heartbeater.cc:511] Master 127.27.174.126:45275 requested a full tablet report, sending...
I20260925 19:13:23.480170 14988 ts_manager.cc:194] Registered new tserver with Master: f5d6506093a8474bb9f5bb911cc7e913 (127.27.174.65:43231)
I20260925 19:13:23.480865 14988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:55145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20260925 19:13:23.496393 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:23.578881 15184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:23.579146 15184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:23.579188 15184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:23.580825 15184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:23.580909 15184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:23.580955 15184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:23.582595 15184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:23.582856 15184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:23.583101 15184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:23.588177 15192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.588326 15190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.588201 15189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:23.588370 15184 server_base.cc:1034] running on GCE node
I20260925 19:13:23.588626 15184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:23.588876 15184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:23.590037 15184 hybrid_clock.cc:648] HybridClock initialized: now 1790363603590002 us; error 39 us; skew 500 ppm
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:23.592670 15184 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20260925 19:13:23.593147 15184 webserver.cc:460] Webserver started at http://127.27.174.66:38713/ using document root <none> and password file <none>
I20260925 19:13:23.593389 15184 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:23.593456 15184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:23.593612 15184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:23.594646 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "e13740a64fde4aca96e01916965240f3"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "9e3e9f9c7f9041583a7d7d27982214a0"
server_key_iv: "c401c48c61933b2a042ecb618fa0fbef"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.595009 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "e13740a64fde4aca96e01916965240f3"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "9e3e9f9c7f9041583a7d7d27982214a0"
server_key_iv: "c401c48c61933b2a042ecb618fa0fbef"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.596395 15184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:13:23.597106 15199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.597391 15184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:23.597484 15184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "e13740a64fde4aca96e01916965240f3"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "9e3e9f9c7f9041583a7d7d27982214a0"
server_key_iv: "c401c48c61933b2a042ecb618fa0fbef"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.597565 15184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:23.625167 15184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:23.627410 15184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:23.627621 15184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:23.627894 15184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:23.628230 15184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:23.628293 15184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.628332 15184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:23.628386 15184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.634732 15184 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:35999
I20260925 19:13:23.634778 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:35999 every 8 connection(s)
I20260925 19:13:23.635118 15184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:23.641492 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15184
I20260925 19:13:23.641621 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:23.641832 28345 external_mini_cluster.cc:1428] Setting key b414b5b655ba6b721057570db2083e8a
I20260925 19:13:23.644955 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.635823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45275 (local address 127.27.174.66:35945)
0925 19:13:23.635978 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.636027 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.636507 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.636729 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.636732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.637291 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:23.637478 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.637483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.638196 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.638196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.638982 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.638983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.639382 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.639382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.639478 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.640027 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:23.640035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:23.641511 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:23.643320 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.643327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.643329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.643721 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.643928 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.643929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.643929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.644114 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.644457 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:23.644463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:23.644601 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.644687 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.644802 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 19:13:23.644968 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.635864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:35945 (local address 127.27.174.126:45275)
0925 19:13:23.635930 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.635932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.636126 (+ 194us) server_negotiation.cc:407] Connection header received
0925 19:13:23.636601 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.636603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.636614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.636679 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:23.637577 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.638103 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.639077 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.639304 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.639367 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.641616 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:23.641618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:23.641618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:23.641627 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:23.643233 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.643805 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.643806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.643806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.643860 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.644195 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.644196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.644197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.644354 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:23.644484 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.644765 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.644859 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:13:23.645607 15313 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45275
I20260925 19:13:23.645706 15313 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:23.645915 15313 heartbeater.cc:511] Master 127.27.174.126:45275 requested a full tablet report, sending...
I20260925 19:13:23.646317 14988 ts_manager.cc:194] Registered new tserver with Master: e13740a64fde4aca96e01916965240f3 (127.27.174.66:35999)
I20260925 19:13:23.646790 14988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:35945
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20260925 19:13:23.668793 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:23.750413 15318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:23.750609 15318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:23.750633 15318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:23.751834 15318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:23.751874 15318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:23.751906 15318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:23.753351 15318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:23.753587 15318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:23.753804 15318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:23.758714 15318 server_base.cc:1034] running on GCE node
W20260925 19:13:23.758671 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.758697 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:23.758730 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:23.759128 15318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:23.759366 15318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:23.760522 15318 hybrid_clock.cc:648] HybridClock initialized: now 1790363603760533 us; error 50 us; skew 500 ppm
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:23.762964 15318 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20260925 19:13:23.763404 15318 webserver.cc:460] Webserver started at http://127.27.174.67:44335/ using document root <none> and password file <none>
I20260925 19:13:23.763674 15318 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:23.763721 15318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:23.763875 15318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:23.764875 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "db3a42a8a461439c8508aa82071aa621"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "933b695ad37513c6a860dbd8e18e88b7"
server_key_iv: "582654179a21857353b68645e26a6e3a"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.765232 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "db3a42a8a461439c8508aa82071aa621"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "933b695ad37513c6a860dbd8e18e88b7"
server_key_iv: "582654179a21857353b68645e26a6e3a"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.766480 15318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:23.767217 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.767398 15318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:23.767490 15318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "db3a42a8a461439c8508aa82071aa621"
format_stamp: "Formatted at 2026-09-25 19:13:23 on dist-test-slave-fcdn"
server_key: "933b695ad37513c6a860dbd8e18e88b7"
server_key_iv: "582654179a21857353b68645e26a6e3a"
server_key_version: "encryptionkey@0"
I20260925 19:13:23.767575 15318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:23.786393 15318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:23.788718 15318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:23.788877 15318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:23.789116 15318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:23.789510 15318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:23.789572 15318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.789608 15318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:23.789629 15318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:23.796470 15318 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42735
I20260925 19:13:23.796520 15446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42735 every 8 connection(s)
I20260925 19:13:23.796818 15318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:13:23 dist-test-slave-fcdn krb5kdc[14943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363603, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:23.803937 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15318
I20260925 19:13:23.804030 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:23.804237 28345 external_mini_cluster.cc:1428] Setting key b9114370f95f39ec824af1f2cba4a29d
I20260925 19:13:23.806098 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.797491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37839 (local address 127.27.174.126:45275)
0925 19:13:23.797537 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.797539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.797692 (+ 153us) server_negotiation.cc:407] Connection header received
0925 19:13:23.798203 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.798204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.798216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.798290 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:23.799186 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.799823 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.800977 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.801170 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.801234 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.803331 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:23.803332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:23.803333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:23.803338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:23.804964 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.805373 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.805374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.805374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.805442 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.805620 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.805621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.805621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.805744 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:23.805786 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.805980 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.806037 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:13:23.806324 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.797433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:45275 (local address 127.27.174.67:37839)
0925 19:13:23.797581 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.797621 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.798109 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.798344 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.798347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.798895 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:23.799093 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.799098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.799904 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.799905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.800894 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.800896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.801249 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.801250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.801346 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.801914 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:23.801922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:23.803248 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:23.805057 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.805064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.805067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.805312 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.805498 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:23.805498 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:23.805498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:23.805584 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:23.805784 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:23.805786 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:23.805888 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.805957 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.806075 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260925 19:13:23.806818 15447 heartbeater.cc:347] Connected to a master server at 127.27.174.126:45275
I20260925 19:13:23.806916 15447 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:23.807070 15447 heartbeater.cc:511] Master 127.27.174.126:45275 requested a full tablet report, sending...
I20260925 19:13:23.807402 14988 ts_manager.cc:194] Registered new tserver with Master: db3a42a8a461439c8508aa82071aa621 (127.27.174.67:42735)
I20260925 19:13:23.807796 14988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:37839
I20260925 19:13:23.816112 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:23.823416 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.817355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.27.174.126:45275)
0925 19:13:23.817414 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.817417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.817543 (+ 126us) server_negotiation.cc:407] Connection header received
0925 19:13:23.817619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.817620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.817631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.817685 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:23.818269 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.818742 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.819476 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.819746 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.819828 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.820902 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:23.820904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:23.820904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:23.820912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:23.822285 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.822618 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.822619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.822619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.822673 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:23.822872 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:23.822873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:23.822873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:23.823020 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:23.823122 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.823231 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.823309 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:23.825031 14988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:23.826665 14988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:23.833164 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:35999 (local address 127.0.0.1:57996)
0925 19:13:23.829185 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.829224 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.829291 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.829598 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.829601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.829603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:23.829668 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.829669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.831047 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.831048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.832018 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.832020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.832857 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.832931 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.832947 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.833023 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.833078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260925 19:13:23.833314 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.27.174.65:43231)
0925 19:13:23.829784 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.829788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.829798 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:13:23.829838 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.829840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.829856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.829948 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:23.830161 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.830832 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832386 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.832924 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832999 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.833057 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.833175 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.833223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":72,"threads_started":1}
I20260925 19:13:23.833501 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.27.174.67:42735)
0925 19:13:23.829395 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.829401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.829437 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:13:23.829519 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.829521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.829535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.829604 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:23.829782 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.830911 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832377 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.833090 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.833168 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.833219 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.833380 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.833421 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260925 19:13:23.833782 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.27.174.66:35999)
0925 19:13:23.829436 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:23.829440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:23.829452 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:13:23.829495 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:23.829498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:23.829515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:23.829597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:23.829761 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.830941 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832104 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.832764 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.832842 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.832895 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:23.833685 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 19:13:23.833723 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 19:13:23.833271 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:43231 (local address 127.0.0.1:35212)
0925 19:13:23.829460 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.829471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.829534 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.830037 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.830038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.830040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:23.830081 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.830083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.830931 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.830932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.832284 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.832286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.833022 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.833023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.833085 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.833090 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.833152 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.833195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":60,"thread_start_us":83,"threads_started":1}
I20260925 19:13:23.833453 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:23.829041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42735 (local address 127.0.0.1:47310)
0925 19:13:23.829390 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:23.829399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:23.829446 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:23.829664 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:23.829664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:23.829666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:23.829707 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.829708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.831009 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.831010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.832285 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:23.832286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:23.833244 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:23.833245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:23.833306 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:23.833310 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:23.833365 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:13:23.833400 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":84,"thread_start_us":71,"threads_started":1}
I20260925 19:13:23.834553 15247 tablet_service.cc:1467] Processing CreateTablet for tablet 6f189455c34640b394bd3fd5f140f856 (DEFAULT_TABLE table=test-table [id=63272d91fd364464bccc7d14626e7ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:23.834570 15113 tablet_service.cc:1467] Processing CreateTablet for tablet 6f189455c34640b394bd3fd5f140f856 (DEFAULT_TABLE table=test-table [id=63272d91fd364464bccc7d14626e7ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:23.834909 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f189455c34640b394bd3fd5f140f856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:23.834909 15247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f189455c34640b394bd3fd5f140f856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:23.834851 15365 tablet_service.cc:1467] Processing CreateTablet for tablet 6f189455c34640b394bd3fd5f140f856 (DEFAULT_TABLE table=test-table [id=63272d91fd364464bccc7d14626e7ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:23.835171 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f189455c34640b394bd3fd5f140f856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:23.837692 15464 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Bootstrap starting.
I20260925 19:13:23.838346 15465 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Bootstrap starting.
I20260925 19:13:23.838382 15464 tablet_bootstrap.cc:654] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:23.838855 15465 tablet_bootstrap.cc:654] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:23.838867 15464 log.cc:824] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:23.839315 15465 log.cc:824] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:23.839577 15464 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: No bootstrap required, opened a new log
I20260925 19:13:23.839627 15466 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Bootstrap starting.
I20260925 19:13:23.839676 15464 ts_tablet_manager.cc:1397] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:23.840034 15465 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: No bootstrap required, opened a new log
I20260925 19:13:23.840103 15465 ts_tablet_manager.cc:1397] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:23.840277 15466 tablet_bootstrap.cc:654] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:23.840762 15466 log.cc:824] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:23.841485 15466 tablet_bootstrap.cc:492] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: No bootstrap required, opened a new log
I20260925 19:13:23.841557 15466 ts_tablet_manager.cc:1397] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:23.841629 15464 raft_consensus.cc:348] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.841810 15464 raft_consensus.cc:374] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:23.841851 15464 raft_consensus.cc:729] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3a42a8a461439c8508aa82071aa621, State: Initialized, Role: FOLLOWER
I20260925 19:13:23.841975 15464 consensus_queue.cc:260] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.842242 15464 ts_tablet_manager.cc:1428] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:13:23.842402 15447 heartbeater.cc:503] Master 127.27.174.126:45275 was elected leader, sending a full tablet report...
I20260925 19:13:23.842357 15465 raft_consensus.cc:348] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.842494 15465 raft_consensus.cc:374] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:23.842518 15465 raft_consensus.cc:729] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d6506093a8474bb9f5bb911cc7e913, State: Initialized, Role: FOLLOWER
I20260925 19:13:23.842584 15465 consensus_queue.cc:260] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.842814 15465 ts_tablet_manager.cc:1428] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:23.842824 15179 heartbeater.cc:503] Master 127.27.174.126:45275 was elected leader, sending a full tablet report...
I20260925 19:13:23.843470 15466 raft_consensus.cc:348] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.843662 15466 raft_consensus.cc:374] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:23.843716 15466 raft_consensus.cc:729] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13740a64fde4aca96e01916965240f3, State: Initialized, Role: FOLLOWER
I20260925 19:13:23.843822 15466 consensus_queue.cc:260] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:23.844141 15466 ts_tablet_manager.cc:1428] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 19:13:23.844159 15313 heartbeater.cc:503] Master 127.27.174.126:45275 was elected leader, sending a full tablet report...
W20260925 19:13:23.885708 15314 tablet.cc:2369] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:13:23.969292 15180 tablet.cc:2369] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:24.037591 15472 raft_consensus.cc:482] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:24.037730 15472 raft_consensus.cc:504] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:24.038108 15472 leader_election.cc:288] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db3a42a8a461439c8508aa82071aa621 (127.27.174.67:42735), f5d6506093a8474bb9f5bb911cc7e913 (127.27.174.65:43231)
I20260925 19:13:24.041402 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.038275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:43231 (local address 127.27.174.66:48103)
0925 19:13:24.038331 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.038345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.038395 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.038580 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.038581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.038583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.038659 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.038661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.039294 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.039295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.040351 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.040353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.041080 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.041081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.041186 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.041189 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.041275 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.041323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 19:13:24.041436 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.038364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48103 (local address 127.27.174.65:43231)
0925 19:13:24.038413 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.038415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.038426 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:24.038460 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.038461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.038470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.038571 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.038735 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.039209 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.040471 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.040847 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.041078 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.041092 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.041306 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.041354 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:13:24.041512 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.038290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42735 (local address 127.27.174.66:34843)
0925 19:13:24.038557 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.038565 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.038629 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.038755 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.038756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.038757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.038835 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.038836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.039452 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.039453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.040662 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.040664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.041257 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.041258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.041333 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.041337 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.041393 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.041434 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":72,"thread_start_us":141,"threads_started":1}
I20260925 19:13:24.041524 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.038400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:34843 (local address 127.27.174.67:42735)
0925 19:13:24.038445 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.038447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.038624 (+ 177us) server_negotiation.cc:407] Connection header received
0925 19:13:24.038685 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.038686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.038696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.038752 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.038913 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.039368 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.040774 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.041169 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.041258 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.041267 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.041420 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.041463 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:24.041770 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f189455c34640b394bd3fd5f140f856" candidate_uuid: "e13740a64fde4aca96e01916965240f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d6506093a8474bb9f5bb911cc7e913" is_pre_election: true
I20260925 19:13:24.041824 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f189455c34640b394bd3fd5f140f856" candidate_uuid: "e13740a64fde4aca96e01916965240f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3a42a8a461439c8508aa82071aa621" is_pre_election: true
I20260925 19:13:24.041952 15401 raft_consensus.cc:2454] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13740a64fde4aca96e01916965240f3 in term 0.
I20260925 19:13:24.041960 15133 raft_consensus.cc:2454] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13740a64fde4aca96e01916965240f3 in term 0.
I20260925 19:13:24.042189 15201 leader_election.cc:302] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db3a42a8a461439c8508aa82071aa621, e13740a64fde4aca96e01916965240f3; no voters:
I20260925 19:13:24.042318 15472 raft_consensus.cc:2781] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:24.042373 15472 raft_consensus.cc:482] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:24.042413 15472 raft_consensus.cc:3037] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.043063 15472 raft_consensus.cc:504] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:24.043203 15472 leader_election.cc:288] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [CANDIDATE]: Term 1 election: Requested vote from peers db3a42a8a461439c8508aa82071aa621 (127.27.174.67:42735), f5d6506093a8474bb9f5bb911cc7e913 (127.27.174.65:43231)
I20260925 19:13:24.043421 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f189455c34640b394bd3fd5f140f856" candidate_uuid: "e13740a64fde4aca96e01916965240f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d6506093a8474bb9f5bb911cc7e913"
I20260925 19:13:24.043507 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f189455c34640b394bd3fd5f140f856" candidate_uuid: "e13740a64fde4aca96e01916965240f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3a42a8a461439c8508aa82071aa621"
I20260925 19:13:24.043519 15133 raft_consensus.cc:3037] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.043619 15401 raft_consensus.cc:3037] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.044361 15133 raft_consensus.cc:2454] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13740a64fde4aca96e01916965240f3 in term 1.
I20260925 19:13:24.044364 15401 raft_consensus.cc:2454] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13740a64fde4aca96e01916965240f3 in term 1.
I20260925 19:13:24.044581 15200 leader_election.cc:302] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e13740a64fde4aca96e01916965240f3, f5d6506093a8474bb9f5bb911cc7e913; no voters:
I20260925 19:13:24.044680 15472 raft_consensus.cc:2781] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:24.044819 15472 raft_consensus.cc:686] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 1 LEADER]: Becoming Leader. State: Replica: e13740a64fde4aca96e01916965240f3, State: Running, Role: LEADER
I20260925 19:13:24.044922 15472 consensus_queue.cc:237] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } }
I20260925 19:13:24.045742 14988 catalog_manager.cc:5495] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13740a64fde4aca96e01916965240f3 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "e13740a64fde4aca96e01916965240f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13740a64fde4aca96e01916965240f3" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 35999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 } health_report { overall_health: UNKNOWN } } }
W20260925 19:13:24.047358 15448 tablet.cc:2369] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:24.059844 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.056959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.27.174.66:35999)
0925 19:13:24.057016 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.057018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.057046 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:13:24.057118 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.057118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.057135 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.057193 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:24.057312 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.057761 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.058859 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.059179 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.059276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.059472 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:24.059582 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:24.059696 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.059727 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.059747 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:13:24.061656 15401 raft_consensus.cc:1261] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 1 FOLLOWER]: Refusing update from remote peer e13740a64fde4aca96e01916965240f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:24.061676 15133 raft_consensus.cc:1261] T 6f189455c34640b394bd3fd5f140f856 P f5d6506093a8474bb9f5bb911cc7e913 [term 1 FOLLOWER]: Refusing update from remote peer e13740a64fde4aca96e01916965240f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:24.061969 15474 consensus_queue.cc:1035] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3a42a8a461439c8508aa82071aa621" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:24.062124 15472 consensus_queue.cc:1035] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d6506093a8474bb9f5bb911cc7e913" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:24.064364 15477 mvcc.cc:205] Tried to move back new op lower bound from 7333329322233982976 to 7333329322168619008. Current Snapshot: MvccSnapshot[applied={T|T < 7333329322233982976}]
I20260925 19:13:24.064553 15478 mvcc.cc:205] Tried to move back new op lower bound from 7333329322233982976 to 7333329322168619008. Current Snapshot: MvccSnapshot[applied={T|T < 7333329322233982976}]
I20260925 19:13:24.065050 15479 mvcc.cc:205] Tried to move back new op lower bound from 7333329322233982976 to 7333329322168619008. Current Snapshot: MvccSnapshot[applied={T|T < 7333329322233982976}]
I20260925 19:13:24.067282 14987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:24.067360 14987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:24.068071 14987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b122816faad641ce8a7290ab29e60cfa: Sending DeleteTablet for 3 replicas of tablet 6f189455c34640b394bd3fd5f140f856
I20260925 19:13:24.068414 15247 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f189455c34640b394bd3fd5f140f856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57996
I20260925 19:13:24.068418 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f189455c34640b394bd3fd5f140f856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47310
I20260925 19:13:24.068436 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f189455c34640b394bd3fd5f140f856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35212
I20260925 19:13:24.068662 15489 tablet_replica.cc:330] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: stopping tablet replica
I20260925 19:13:24.068756 15489 raft_consensus.cc:2229] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:13:24.068872 15487 tablet_replica.cc:330] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: stopping tablet replica
I20260925 19:13:24.069069 15487 raft_consensus.cc:2229] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:24.069118 15489 raft_consensus.cc:2258] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:24.069795 15489 ts_tablet_manager.cc:1905] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:24.069061 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15050
I20260925 19:13:24.070343 15487 raft_consensus.cc:2258] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:24.070809 15487 ts_tablet_manager.cc:1905] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:24.071758 15489 ts_tablet_manager.cc:1918] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:24.071832 15489 log.cc:1196] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/6f189455c34640b394bd3fd5f140f856
I20260925 19:13:24.072118 15489 ts_tablet_manager.cc:1939] T 6f189455c34640b394bd3fd5f140f856 P e13740a64fde4aca96e01916965240f3: Deleting consensus metadata
I20260925 19:13:24.072511 14975 catalog_manager.cc:4841] TS e13740a64fde4aca96e01916965240f3 (127.27.174.66:35999): tablet 6f189455c34640b394bd3fd5f140f856 (table test-table [id=63272d91fd364464bccc7d14626e7ef0]) successfully deleted
I20260925 19:13:24.072870 15487 ts_tablet_manager.cc:1918] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:24.072957 15487 log.cc:1196] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/6f189455c34640b394bd3fd5f140f856
I20260925 19:13:24.073207 15487 ts_tablet_manager.cc:1939] T 6f189455c34640b394bd3fd5f140f856 P db3a42a8a461439c8508aa82071aa621: Deleting consensus metadata
I20260925 19:13:24.073650 14973 catalog_manager.cc:4841] TS db3a42a8a461439c8508aa82071aa621 (127.27.174.67:42735): tablet 6f189455c34640b394bd3fd5f140f856 (table test-table [id=63272d91fd364464bccc7d14626e7ef0]) successfully deleted
W20260925 19:13:24.077273 14972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:43231 (error 108)
I20260925 19:13:24.077814 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.077680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:43231 (local address 127.0.0.1:35228)
0925 19:13:24.077733 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.077772 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:13:24.077852 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15184
W20260925 19:13:24.078671 14972 catalog_manager.cc:4584] TS f5d6506093a8474bb9f5bb911cc7e913 (127.27.174.65:43231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f189455c34640b394bd3fd5f140f856: Network error: Client connection negotiation failed: client connection to 127.27.174.65:43231: connect: Connection refused (error 111)
I20260925 19:13:24.082561 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15318
I20260925 19:13:24.087185 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 14956
2026-09-25T19:13:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2110 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T19:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:24Z Disabled control of system clock
I20260925 19:13:24.115476 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41463
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:24.198046 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:24.198297 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:24.198319 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:24.199656 15494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:24.199716 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:24.199729 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:24.199743 15494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:24.199755 15494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:24.201164 15494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:24.201380 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:24.201654 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:24.207026 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.207005 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.207010 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:24.207083 15494 server_base.cc:1034] running on GCE node
I20260925 19:13:24.207435 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:24.207834 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:24.209003 15494 hybrid_clock.cc:648] HybridClock initialized: now 1790363604208997 us; error 47 us; skew 500 ppm
I20260925 19:13:24.210377 15494 webserver.cc:460] Webserver started at http://127.27.174.126:35611/ using document root <none> and password file <none>
I20260925 19:13:24.210652 15494 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:24.210723 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:24.210879 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:24.212075 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "4c9828c791fa4aa585f726e3012c2a3d"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "1bb118d11e76bb874c4e9ec5f9b42407"
server_key_iv: "d46e321438d821f38b9299e5dbeef839"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.212451 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "4c9828c791fa4aa585f726e3012c2a3d"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "1bb118d11e76bb874c4e9ec5f9b42407"
server_key_iv: "d46e321438d821f38b9299e5dbeef839"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.213855 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:24.214736 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.214927 15494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:24.215036 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "4c9828c791fa4aa585f726e3012c2a3d"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "1bb118d11e76bb874c4e9ec5f9b42407"
server_key_iv: "d46e321438d821f38b9299e5dbeef839"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.215113 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:24.233023 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:24.233444 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:24.233629 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:24.237778 15494 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41463
I20260925 19:13:24.237814 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41463 every 8 connection(s)
I20260925 19:13:24.238180 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:24.239224 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:24.240130 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15494
I20260925 19:13:24.240211 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:24.240408 28345 external_mini_cluster.cc:1428] Setting key 319b32fb345c91ad6664b4efd39e0e2d
I20260925 19:13:24.242506 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: Bootstrap starting.
I20260925 19:13:24.243165 15561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:24.243471 15561 log.cc:824] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:24.244251 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: No bootstrap required, opened a new log
I20260925 19:13:24.246083 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.240973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.27.174.126:41463)
0925 19:13:24.241176 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.241183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.241209 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:13:24.241639 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.241645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.241714 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.241831 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:24.242123 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.243005 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.244092 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.244543 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.244639 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.245384 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:24.245387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:24.245388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:24.245550 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:24.245583 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:24.245594 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:24.245655 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.245760 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.245854 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260925 19:13:24.246644 15561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } }
I20260925 19:13:24.246845 15561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:24.246893 15561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9828c791fa4aa585f726e3012c2a3d, State: Initialized, Role: FOLLOWER
I20260925 19:13:24.247016 15561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } }
I20260925 19:13:24.247090 15561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:24.247112 15561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:24.247161 15561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.247819 15561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } }
I20260925 19:13:24.247910 15561 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9828c791fa4aa585f726e3012c2a3d; no voters:
I20260925 19:13:24.248142 15561 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:24.248267 15566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:24.248423 15561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:24.248453 15566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [term 1 LEADER]: Becoming Leader. State: Replica: 4c9828c791fa4aa585f726e3012c2a3d, State: Running, Role: LEADER
I20260925 19:13:24.248567 15566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } }
I20260925 19:13:24.248986 15568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9828c791fa4aa585f726e3012c2a3d. Latest consensus state: current_term: 1 leader_uuid: "4c9828c791fa4aa585f726e3012c2a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } } }
I20260925 19:13:24.248984 15567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9828c791fa4aa585f726e3012c2a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9828c791fa4aa585f726e3012c2a3d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41463 } } }
I20260925 19:13:24.249092 15568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:24.249096 15567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:24.249897 15575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:24.250582 15575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:24.252140 15575 catalog_manager.cc:1347] Generated new cluster ID: aa2a29f144604389b592bc7e409ecbcd
I20260925 19:13:24.252204 15575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:24.280004 15575 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:24.280720 15575 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:24.290339 15575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: Generated new TSK 0
I20260925 19:13:24.290565 15575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:24.298365 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--builtin_ntp_servers=127.27.174.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:24.386363 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:24.386623 15585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:24.386663 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:24.388049 15585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:24.388116 15585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:24.388159 15585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20260925 19:13:24.389657 15585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:24.389997 15585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:24.390348 15585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:24.395888 15593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.395888 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.396311 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:24.396868 15585 server_base.cc:1034] running on GCE node
I20260925 19:13:24.397143 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:24.397447 15585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:24.398617 15585 hybrid_clock.cc:648] HybridClock initialized: now 1790363604398603 us; error 26 us; skew 500 ppm
I20260925 19:13:24.400100 15585 webserver.cc:460] Webserver started at http://127.27.174.65:39235/ using document root <none> and password file <none>
I20260925 19:13:24.400367 15585 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:24.400439 15585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:24.400599 15585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:24.401744 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data/instance:
uuid: "95ae34964b5a4e4185ed90c5a495f9cf"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "a6823089445725acbdaadf686ce05bd5"
server_key_iv: "f3a2a83d4fed460daa9a45c7f4d18919"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.402151 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "95ae34964b5a4e4185ed90c5a495f9cf"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "a6823089445725acbdaadf686ce05bd5"
server_key_iv: "f3a2a83d4fed460daa9a45c7f4d18919"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.403725 15585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 19:13:24.404524 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.404697 15585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:24.404786 15585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal
uuid: "95ae34964b5a4e4185ed90c5a495f9cf"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "a6823089445725acbdaadf686ce05bd5"
server_key_iv: "f3a2a83d4fed460daa9a45c7f4d18919"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.404888 15585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:24.415751 15585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:24.416106 15585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:24.416272 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:24.416559 15585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:24.416975 15585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:24.417033 15585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.417071 15585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:24.417092 15585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.424888 15585 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35111
I20260925 19:13:24.424916 15712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35111 every 8 connection(s)
I20260925 19:13:24.425383 15585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/data/info.pb
I20260925 19:13:24.431486 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.426294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43249 (local address 127.27.174.126:41463)
0925 19:13:24.426342 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.426345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.426695 (+ 350us) server_negotiation.cc:407] Connection header received
0925 19:13:24.427272 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.427274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.427292 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.427344 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:24.428389 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.428942 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.429964 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.430178 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.430260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.431179 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:24.431180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:24.431180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:24.431185 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:24.431200 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:24.431205 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:24.431255 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.431366 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.431400 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:13:24.431870 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.426315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41463 (local address 127.27.174.65:43249)
0925 19:13:24.426536 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.426580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.427138 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.427537 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.427542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.427553 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:24.428252 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.428263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.429054 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.429054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.429849 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.429852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.430270 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.430271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.430413 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.431014 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:24.431014 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:24.431050 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:24.431076 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:24.431273 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:24.431278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:24.431282 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.431369 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.431479 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
I20260925 19:13:24.432372 15713 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41463
I20260925 19:13:24.432497 15713 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:24.432691 15713 heartbeater.cc:511] Master 127.27.174.126:41463 requested a full tablet report, sending...
I20260925 19:13:24.433243 15525 ts_manager.cc:194] Registered new tserver with Master: 95ae34964b5a4e4185ed90c5a495f9cf (127.27.174.65:35111)
I20260925 19:13:24.433779 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15585
I20260925 19:13:24.433873 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-0/wal/instance
I20260925 19:13:24.434053 28345 external_mini_cluster.cc:1428] Setting key 8ca81aa36e7d0f869780f54246ca71ff
I20260925 19:13:24.434582 15525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:43249
I20260925 19:13:24.435747 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--builtin_ntp_servers=127.27.174.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:13:24.521659 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:24.521939 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:24.521982 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:24.523240 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:24.523312 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:24.523356 15716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20260925 19:13:24.524787 15716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:24.525079 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:24.525399 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:24.530781 15721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.530745 15722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.530754 15724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:24.531158 15716 server_base.cc:1034] running on GCE node
I20260925 19:13:24.531421 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:24.531728 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:24.532886 15716 hybrid_clock.cc:648] HybridClock initialized: now 1790363604532872 us; error 33 us; skew 500 ppm
I20260925 19:13:24.534328 15716 webserver.cc:460] Webserver started at http://127.27.174.66:37271/ using document root <none> and password file <none>
I20260925 19:13:24.534602 15716 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:24.534678 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:24.534834 15716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:24.536036 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data/instance:
uuid: "722780a5164a44a8a0d01ee745e097d4"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "ed27583913e6b4bed92f954285c02754"
server_key_iv: "5e9a6afcf4fa5dd3f77a18eafff2bc0e"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.536407 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "722780a5164a44a8a0d01ee745e097d4"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "ed27583913e6b4bed92f954285c02754"
server_key_iv: "5e9a6afcf4fa5dd3f77a18eafff2bc0e"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.537827 15716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:24.538623 15730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.538795 15716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:24.538936 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal
uuid: "722780a5164a44a8a0d01ee745e097d4"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "ed27583913e6b4bed92f954285c02754"
server_key_iv: "5e9a6afcf4fa5dd3f77a18eafff2bc0e"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.539022 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:24.551337 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:24.551739 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:24.551896 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:24.552291 15716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:24.552672 15716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:24.552731 15716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.552774 15716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:24.552801 15716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.558750 15716 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41947
I20260925 19:13:24.558790 15843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41947 every 8 connection(s)
I20260925 19:13:24.559157 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/data/info.pb
I20260925 19:13:24.560797 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15716
I20260925 19:13:24.560901 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal/instance
I20260925 19:13:24.561097 28345 external_mini_cluster.cc:1428] Setting key c70d721339cc9e94f305bf68afea0d7e
I20260925 19:13:24.562052 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--builtin_ntp_servers=127.27.174.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:13:24.565156 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.560005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53795 (local address 127.27.174.126:41463)
0925 19:13:24.560111 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.560113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.560316 (+ 203us) server_negotiation.cc:407] Connection header received
0925 19:13:24.561107 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.561108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.561121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.561212 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:24.561964 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.562554 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.563631 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.563836 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.563927 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.564836 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:24.564837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:24.564838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:24.564842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:24.564857 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:24.564861 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:24.564934 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.565039 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.565074 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:13:24.565372 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.559936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41463 (local address 127.27.174.66:53795)
0925 19:13:24.560194 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.560229 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.560900 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.561340 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.561344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.561353 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:24.561862 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.561867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.562650 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.562651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.563480 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.563483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.563938 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.563939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.564086 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.564668 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:24.564669 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:24.564699 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:24.564725 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:24.564934 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:24.564940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:24.564944 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.565035 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.565148 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260925 19:13:24.565864 15844 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41463
I20260925 19:13:24.565948 15844 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:24.566159 15844 heartbeater.cc:511] Master 127.27.174.126:41463 requested a full tablet report, sending...
I20260925 19:13:24.566566 15525 ts_manager.cc:194] Registered new tserver with Master: 722780a5164a44a8a0d01ee745e097d4 (127.27.174.66:41947)
I20260925 19:13:24.567154 15525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:53795
W20260925 19:13:24.645133 15847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:24.645346 15847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:24.645368 15847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:24.646620 15847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:24.646663 15847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:24.646689 15847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20260925 19:13:24.648159 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:24.648416 15847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:24.648780 15847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:24.654235 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:24.654244 15847 server_base.cc:1034] running on GCE node
W20260925 19:13:24.654335 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:24.654233 15855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:24.654784 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:24.655050 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:24.656215 15847 hybrid_clock.cc:648] HybridClock initialized: now 1790363604656205 us; error 29 us; skew 500 ppm
I20260925 19:13:24.657626 15847 webserver.cc:460] Webserver started at http://127.27.174.67:38615/ using document root <none> and password file <none>
I20260925 19:13:24.657886 15847 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:24.657954 15847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:24.658114 15847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:24.659214 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data/instance:
uuid: "c75d24732ea345d6901cfd16fc4492e0"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "9c302bcbb66e9ada8123476d8eb62eec"
server_key_iv: "6e2c1f46a759fecf9b1c423b24402821"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.659644 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "c75d24732ea345d6901cfd16fc4492e0"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "9c302bcbb66e9ada8123476d8eb62eec"
server_key_iv: "6e2c1f46a759fecf9b1c423b24402821"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.660997 15847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:24.662087 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.662325 15847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:24.662469 15847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal
uuid: "c75d24732ea345d6901cfd16fc4492e0"
format_stamp: "Formatted at 2026-09-25 19:13:24 on dist-test-slave-fcdn"
server_key: "9c302bcbb66e9ada8123476d8eb62eec"
server_key_iv: "6e2c1f46a759fecf9b1c423b24402821"
server_key_version: "encryptionkey@0"
I20260925 19:13:24.662550 15847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:24.676069 15847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:24.676380 15847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:24.676520 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:24.676757 15847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:13:24.677114 15847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:13:24.677167 15847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.677191 15847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:13:24.677203 15847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:24.682857 15847 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:40259
I20260925 19:13:24.682901 15974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:40259 every 8 connection(s)
I20260925 19:13:24.683233 15847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/data/info.pb
I20260925 19:13:24.687112 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 15847
I20260925 19:13:24.687232 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal/instance
I20260925 19:13:24.687419 28345 external_mini_cluster.cc:1428] Setting key b61a01e19c44b0f0ab096d47a49c04c6
I20260925 19:13:24.688735 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.683997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:39119 (local address 127.27.174.126:41463)
0925 19:13:24.684081 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.684083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.684301 (+ 218us) server_negotiation.cc:407] Connection header received
0925 19:13:24.684792 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.684793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.684803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.684867 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:24.685587 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.686073 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.687156 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.687466 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.687539 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.688507 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:24.688508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:24.688509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:24.688514 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:24.688532 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:24.688536 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:24.688591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.688653 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.688675 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:13:24.689018 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.683965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41463 (local address 127.27.174.67:39119)
0925 19:13:24.684173 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.684215 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.684708 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.684975 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.684979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.684987 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:24.685511 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.685517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.686189 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.686190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.687037 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.687040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.687551 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.687552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.687727 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.688332 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:13:24.688333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:24.688372 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:13:24.688404 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:24.688601 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:24.688604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:24.688607 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.688657 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.688739 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260925 19:13:24.689570 15975 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41463
I20260925 19:13:24.689683 15975 heartbeater.cc:464] Registering TS with master...
I20260925 19:13:24.689880 15975 heartbeater.cc:511] Master 127.27.174.126:41463 requested a full tablet report, sending...
I20260925 19:13:24.690289 15525 ts_manager.cc:194] Registered new tserver with Master: c75d24732ea345d6901cfd16fc4492e0 (127.27.174.67:40259)
I20260925 19:13:24.690670 15525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:39119
I20260925 19:13:24.699325 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:13:24.703711 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.700893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.27.174.126:41463)
0925 19:13:24.700953 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.700955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.701019 (+ 64us) server_negotiation.cc:407] Connection header received
0925 19:13:24.701079 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.701080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.701089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.701135 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:24.701284 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.701759 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.702466 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.702647 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.702698 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.703391 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:24.703392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:24.703392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:24.703396 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:24.703407 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:24.703410 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:24.703448 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.703530 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.703549 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:13:24.705464 15525 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:13:24.707072 15525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:13:24.713541 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:40259 (local address 127.0.0.1:35294)
0925 19:13:24.709499 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.709560 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.709630 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.710076 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.710079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.710081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.710152 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.710155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.711456 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.711457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.712551 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.712553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713233 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.713310 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713323 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.713400 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.713454 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":46,"thread_start_us":71,"threads_started":1}
I20260925 19:13:24.713543 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.27.174.67:40259)
0925 19:13:24.709747 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.709750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.709761 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:24.709804 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.709806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.709824 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.709999 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.710230 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.711364 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.712633 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713143 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713233 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713330 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.713413 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.713453 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260925 19:13:24.713769 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.27.174.65:35111)
0925 19:13:24.709734 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.709739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.709886 (+ 147us) server_negotiation.cc:407] Connection header received
0925 19:13:24.709955 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.709958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.709973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.710047 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.710252 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.710963 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.712453 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713197 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713269 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713358 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.713621 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.713670 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260925 19:13:24.713807 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41947 (local address 127.0.0.1:41198)
0925 19:13:24.709561 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.709569 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.709630 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.710241 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.710242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.710244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.710278 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.710279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.711513 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.711514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.712497 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.712499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713430 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.713480 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713483 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.713528 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.713575 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260925 19:13:24.714146 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35111 (local address 127.0.0.1:45896)
0925 19:13:24.709792 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.709802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.709887 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.710117 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.710118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.710120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.710168 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.710170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.711060 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.711061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.712360 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.712363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713394 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.713470 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713475 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.713528 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.713575 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260925 19:13:24.714393 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.709404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.27.174.66:41947)
0925 19:13:24.710025 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.710029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.710039 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:13:24.710082 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.710084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.710097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.710170 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.710351 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.711429 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.712590 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.713277 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.713332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.713403 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.714239 (+ 836us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.714290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":75,"threads_started":1}
I20260925 19:13:24.714519 15909 tablet_service.cc:1467] Processing CreateTablet for tablet c716b8e894a74ccdb09147987b39ecf4 (DEFAULT_TABLE table=test-table [id=8973a9c4ba6c404786f56e77fb9017e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:24.714887 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c716b8e894a74ccdb09147987b39ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:24.714957 15647 tablet_service.cc:1467] Processing CreateTablet for tablet c716b8e894a74ccdb09147987b39ecf4 (DEFAULT_TABLE table=test-table [id=8973a9c4ba6c404786f56e77fb9017e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:24.715047 15778 tablet_service.cc:1467] Processing CreateTablet for tablet c716b8e894a74ccdb09147987b39ecf4 (DEFAULT_TABLE table=test-table [id=8973a9c4ba6c404786f56e77fb9017e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:13:24.715301 15647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c716b8e894a74ccdb09147987b39ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:24.715325 15778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c716b8e894a74ccdb09147987b39ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:24.717761 15989 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Bootstrap starting.
I20260925 19:13:24.718468 15989 tablet_bootstrap.cc:654] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:24.718569 15990 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Bootstrap starting.
I20260925 19:13:24.718945 15989 log.cc:824] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:24.719281 15990 tablet_bootstrap.cc:654] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:24.719626 15990 log.cc:824] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:24.719795 15989 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: No bootstrap required, opened a new log
I20260925 19:13:24.719771 15991 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Bootstrap starting.
I20260925 19:13:24.719869 15989 ts_tablet_manager.cc:1397] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:13:24.720395 15991 tablet_bootstrap.cc:654] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:24.720521 15990 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: No bootstrap required, opened a new log
I20260925 19:13:24.720604 15990 ts_tablet_manager.cc:1397] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:24.720716 15991 log.cc:824] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:24.721379 15991 tablet_bootstrap.cc:492] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: No bootstrap required, opened a new log
I20260925 19:13:24.721469 15991 ts_tablet_manager.cc:1397] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:13:24.722579 15989 raft_consensus.cc:348] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.722764 15989 raft_consensus.cc:374] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:24.722800 15989 raft_consensus.cc:729] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75d24732ea345d6901cfd16fc4492e0, State: Initialized, Role: FOLLOWER
I20260925 19:13:24.722947 15989 consensus_queue.cc:260] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.723224 15989 ts_tablet_manager.cc:1428] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:24.723140 15990 raft_consensus.cc:348] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.723280 15990 raft_consensus.cc:374] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:24.723320 15990 raft_consensus.cc:729] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ae34964b5a4e4185ed90c5a495f9cf, State: Initialized, Role: FOLLOWER
I20260925 19:13:24.723349 15975 heartbeater.cc:503] Master 127.27.174.126:41463 was elected leader, sending a full tablet report...
I20260925 19:13:24.723452 15990 consensus_queue.cc:260] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.723750 15990 ts_tablet_manager.cc:1428] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:24.723910 15713 heartbeater.cc:503] Master 127.27.174.126:41463 was elected leader, sending a full tablet report...
I20260925 19:13:24.723866 15991 raft_consensus.cc:348] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.723999 15991 raft_consensus.cc:374] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:24.724048 15991 raft_consensus.cc:729] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722780a5164a44a8a0d01ee745e097d4, State: Initialized, Role: FOLLOWER
I20260925 19:13:24.724190 15991 consensus_queue.cc:260] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.724428 15991 ts_tablet_manager.cc:1428] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:13:24.724565 15844 heartbeater.cc:503] Master 127.27.174.126:41463 was elected leader, sending a full tablet report...
W20260925 19:13:24.809711 15845 tablet.cc:2369] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:24.889108 15996 raft_consensus.cc:482] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:13:24.889238 15996 raft_consensus.cc:504] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.889703 15996 leader_election.cc:288] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722780a5164a44a8a0d01ee745e097d4 (127.27.174.66:41947), c75d24732ea345d6901cfd16fc4492e0 (127.27.174.67:40259)
I20260925 19:13:24.892725 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.890027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:37679 (local address 127.27.174.66:41947)
0925 19:13:24.890077 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.890079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.890090 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:24.890132 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.890133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.890142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.890222 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.890392 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.890859 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892016 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.892369 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892458 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.892466 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.892615 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.892655 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:13:24.892725 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.889921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41947 (local address 127.27.174.65:37679)
0925 19:13:24.889979 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.889992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.890042 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.890230 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.890231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.890233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.890313 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.890315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.890950 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.890951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.891930 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.891933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.892456 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.892533 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.892537 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.892610 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.892654 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:13:24.892974 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.890052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:55185 (local address 127.27.174.67:40259)
0925 19:13:24.890095 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.890097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.890253 (+ 156us) server_negotiation.cc:407] Connection header received
0925 19:13:24.890320 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.890321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.890330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.890375 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:13:24.890542 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.891023 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892243 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.892609 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892694 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.892701 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.892856 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.892911 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:13:24.893069 15798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c716b8e894a74ccdb09147987b39ecf4" candidate_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722780a5164a44a8a0d01ee745e097d4" is_pre_election: true
I20260925 19:13:24.893179 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.889943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:40259 (local address 127.27.174.65:55185)
0925 19:13:24.890178 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.890187 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:24.890253 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:24.890386 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:24.890387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:24.890389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:13:24.890465 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.890466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.891111 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.891112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.892158 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:24.892160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.892694 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.892695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:24.892764 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.892768 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:13:24.893054 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 19:13:24.893098 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":70,"thread_start_us":115,"threads_started":1}
I20260925 19:13:24.893242 15798 raft_consensus.cc:2454] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ae34964b5a4e4185ed90c5a495f9cf in term 0.
I20260925 19:13:24.893501 15602 leader_election.cc:302] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722780a5164a44a8a0d01ee745e097d4, 95ae34964b5a4e4185ed90c5a495f9cf; no voters:
I20260925 19:13:24.893563 15929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c716b8e894a74ccdb09147987b39ecf4" candidate_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75d24732ea345d6901cfd16fc4492e0" is_pre_election: true
I20260925 19:13:24.893637 15996 raft_consensus.cc:2781] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:13:24.893687 15996 raft_consensus.cc:482] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:13:24.893713 15996 raft_consensus.cc:3037] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.893712 15929 raft_consensus.cc:2454] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ae34964b5a4e4185ed90c5a495f9cf in term 0.
I20260925 19:13:24.894454 15996 raft_consensus.cc:504] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.894603 15996 leader_election.cc:288] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [CANDIDATE]: Term 1 election: Requested vote from peers 722780a5164a44a8a0d01ee745e097d4 (127.27.174.66:41947), c75d24732ea345d6901cfd16fc4492e0 (127.27.174.67:40259)
I20260925 19:13:24.894761 15929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c716b8e894a74ccdb09147987b39ecf4" candidate_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75d24732ea345d6901cfd16fc4492e0"
I20260925 19:13:24.894800 15798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c716b8e894a74ccdb09147987b39ecf4" candidate_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722780a5164a44a8a0d01ee745e097d4"
I20260925 19:13:24.894896 15798 raft_consensus.cc:3037] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.894845 15929 raft_consensus.cc:3037] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:24.895646 15798 raft_consensus.cc:2454] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ae34964b5a4e4185ed90c5a495f9cf in term 1.
I20260925 19:13:24.895777 15929 raft_consensus.cc:2454] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ae34964b5a4e4185ed90c5a495f9cf in term 1.
I20260925 19:13:24.895826 15602 leader_election.cc:302] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722780a5164a44a8a0d01ee745e097d4, 95ae34964b5a4e4185ed90c5a495f9cf; no voters:
I20260925 19:13:24.895912 15996 raft_consensus.cc:2781] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:24.896062 15996 raft_consensus.cc:686] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [term 1 LEADER]: Becoming Leader. State: Replica: 95ae34964b5a4e4185ed90c5a495f9cf, State: Running, Role: LEADER
I20260925 19:13:24.896167 15996 consensus_queue.cc:237] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } }
I20260925 19:13:24.897051 15524 catalog_manager.cc:5495] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 95ae34964b5a4e4185ed90c5a495f9cf (127.27.174.65). New cstate: current_term: 1 leader_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95ae34964b5a4e4185ed90c5a495f9cf" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35111 } health_report { overall_health: HEALTHY } } }
W20260925 19:13:24.926317 15714 tablet.cc:2369] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:13:24.933746 15976 tablet.cc:2369] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:13:24.941136 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.938696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.27.174.65:35111)
0925 19:13:24.938770 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:24.938772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:24.938786 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:13:24.938821 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:24.938822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:24.938840 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:24.938917 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:24.939060 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.939543 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.940331 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:24.940528 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:24.940611 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:24.940784 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:24.940885 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:24.940976 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:24.941009 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 19:13:24.941028 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:13:24.942898 15929 raft_consensus.cc:1261] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 1 FOLLOWER]: Refusing update from remote peer 95ae34964b5a4e4185ed90c5a495f9cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:24.942960 15798 raft_consensus.cc:1261] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 1 FOLLOWER]: Refusing update from remote peer 95ae34964b5a4e4185ed90c5a495f9cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:24.943241 15996 consensus_queue.cc:1035] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75d24732ea345d6901cfd16fc4492e0" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:24.943399 15999 consensus_queue.cc:1035] T c716b8e894a74ccdb09147987b39ecf4 P 95ae34964b5a4e4185ed90c5a495f9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "722780a5164a44a8a0d01ee745e097d4" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:24.946189 16003 mvcc.cc:205] Tried to move back new op lower bound from 7333329325843611648 to 7333329325655310336. Current Snapshot: MvccSnapshot[applied={T|T < 7333329325843611648}]
I20260925 19:13:24.947968 15525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:24.948069 15525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:13:24.948872 15525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c9828c791fa4aa585f726e3012c2a3d: Sending DeleteTablet for 3 replicas of tablet c716b8e894a74ccdb09147987b39ecf4
I20260925 19:13:24.949244 15778 tablet_service.cc:1514] Processing DeleteTablet for tablet c716b8e894a74ccdb09147987b39ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='slave'} at 127.0.0.1:41198
I20260925 19:13:24.949268 15647 tablet_service.cc:1514] Processing DeleteTablet for tablet c716b8e894a74ccdb09147987b39ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='slave'} at 127.0.0.1:45896
I20260925 19:13:24.949481 16012 tablet_replica.cc:330] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: stopping tablet replica
I20260925 19:13:24.949743 16012 raft_consensus.cc:2229] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:24.949719 15909 tablet_service.cc:1514] Processing DeleteTablet for tablet c716b8e894a74ccdb09147987b39ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:13:24 UTC) from {username='slave'} at 127.0.0.1:35294
I20260925 19:13:24.949916 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15585
I20260925 19:13:24.949878 16012 raft_consensus.cc:2258] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:24.951009 16012 ts_tablet_manager.cc:1905] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:24.952822 16012 ts_tablet_manager.cc:1918] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:24.952889 16012 log.cc:1196] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-1/wal/wals/c716b8e894a74ccdb09147987b39ecf4
I20260925 19:13:24.953094 16012 ts_tablet_manager.cc:1939] T c716b8e894a74ccdb09147987b39ecf4 P 722780a5164a44a8a0d01ee745e097d4: Deleting consensus metadata
I20260925 19:13:24.953562 15511 catalog_manager.cc:4841] TS 722780a5164a44a8a0d01ee745e097d4 (127.27.174.66:41947): tablet c716b8e894a74ccdb09147987b39ecf4 (table test-table [id=8973a9c4ba6c404786f56e77fb9017e1]) successfully deleted
W20260925 19:13:24.957436 15509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:35111 (error 108)
I20260925 19:13:24.957479 16013 tablet_replica.cc:330] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: stopping tablet replica
I20260925 19:13:24.957600 16013 raft_consensus.cc:2229] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:13:24.957609 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15716
I20260925 19:13:24.957716 16013 raft_consensus.cc:2258] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:13:24.958153 16013 ts_tablet_manager.cc:1905] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:13:24.958396 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:24.958268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35111 (local address 127.0.0.1:45920)
0925 19:13:24.958321 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:24.958363 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:13:24.958499 15509 catalog_manager.cc:4584] TS 95ae34964b5a4e4185ed90c5a495f9cf (127.27.174.65:35111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c716b8e894a74ccdb09147987b39ecf4: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35111: connect: Connection refused (error 111)
I20260925 19:13:24.959717 16013 ts_tablet_manager.cc:1918] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:13:24.959797 16013 log.cc:1196] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Deleting WAL directory at /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363366557183-28345-0/minicluster-data/ts-2/wal/wals/c716b8e894a74ccdb09147987b39ecf4
I20260925 19:13:24.960017 16013 ts_tablet_manager.cc:1939] T c716b8e894a74ccdb09147987b39ecf4 P c75d24732ea345d6901cfd16fc4492e0: Deleting consensus metadata
I20260925 19:13:24.960413 15512 catalog_manager.cc:4841] TS c75d24732ea345d6901cfd16fc4492e0 (127.27.174.67:40259): tablet c716b8e894a74ccdb09147987b39ecf4 (table test-table [id=8973a9c4ba6c404786f56e77fb9017e1]) successfully deleted
I20260925 19:13:24.962653 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15847
I20260925 19:13:24.967207 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 15494
2026-09-25T19:13:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (878 ms)
[----------] 6 tests from EncryptionPolicyTest (8714 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:25 dist-test-slave-fcdn krb5kdc[16017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:25 dist-test-slave-fcdn krb5kdc[16017](info): set up 2 sockets
Sep 25 19:13:25 dist-test-slave-fcdn krb5kdc[16017](info): commencing operation
krb5kdc: starting...
W20260925 19:13:26.015522 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:26.130817 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:26.212484 16030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:26.212728 16030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:26.212765 16030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:26.213903 16030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:26.213972 16030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:26.213999 16030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:26.214030 16030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:26.214061 16030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:26.215505 16030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:26.215791 16030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:26.216028 16030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:26.220965 16036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:26.221163 16030 server_base.cc:1034] running on GCE node
W20260925 19:13:26.220988 16038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:26.220971 16035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:26.221550 16030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:26.221865 16030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:26.223104 16030 hybrid_clock.cc:648] HybridClock initialized: now 1790363606223099 us; error 51 us; skew 500 ppm
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:26.226374 16030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:26.226780 16030 webserver.cc:460] Webserver started at http://127.0.0.1:33119/ using document root <none> and password file <none>
I20260925 19:13:26.226995 16030 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:26.227038 16030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:26.227130 16030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:26.228286 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "ec417a9961f442e4a53a3f3accc14d40"
format_stamp: "Formatted at 2026-09-25 19:13:26 on dist-test-slave-fcdn"
server_key: "30629f8046a760d63648ab91cae3e7c3"
server_key_iv: "bb4a14090042e011331f16c40947824e"
server_key_version: "encryptionkey@0"
I20260925 19:13:26.228607 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ec417a9961f442e4a53a3f3accc14d40"
format_stamp: "Formatted at 2026-09-25 19:13:26 on dist-test-slave-fcdn"
server_key: "30629f8046a760d63648ab91cae3e7c3"
server_key_iv: "bb4a14090042e011331f16c40947824e"
server_key_version: "encryptionkey@0"
I20260925 19:13:26.229944 16030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:26.230639 16045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:26.230880 16030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:26.230966 16030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "ec417a9961f442e4a53a3f3accc14d40"
format_stamp: "Formatted at 2026-09-25 19:13:26 on dist-test-slave-fcdn"
server_key: "30629f8046a760d63648ab91cae3e7c3"
server_key_iv: "bb4a14090042e011331f16c40947824e"
server_key_version: "encryptionkey@0"
I20260925 19:13:26.231020 16030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:26.239415 16030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:26.241216 16030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:26.241317 16030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:26.245031 16030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45377
I20260925 19:13:26.245079 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45377 every 8 connection(s)
I20260925 19:13:26.245407 16030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:26.246292 16098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:26.248723 16098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40: Bootstrap starting.
I20260925 19:13:26.249315 16098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:26.249615 16098 log.cc:824] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:26.250289 16098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40: No bootstrap required, opened a new log
I20260925 19:13:26.252051 16098 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 19:13:26.252213 16098 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:26.252264 16098 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec417a9961f442e4a53a3f3accc14d40, State: Initialized, Role: FOLLOWER
I20260925 19:13:26.252424 16098 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 19:13:26.252513 16098 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:26.252560 16098 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:26.252614 16098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:26.253175 16098 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 19:13:26.253297 16098 leader_election.cc:302] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec417a9961f442e4a53a3f3accc14d40; no voters:
I20260925 19:13:26.253486 16098 leader_election.cc:288] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:26.253510 16101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:26.253665 16101 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [term 1 LEADER]: Becoming Leader. State: Replica: ec417a9961f442e4a53a3f3accc14d40, State: Running, Role: LEADER
I20260925 19:13:26.253748 16098 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:26.253821 16101 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 19:13:26.254220 16103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec417a9961f442e4a53a3f3accc14d40. Latest consensus state: current_term: 1 leader_uuid: "ec417a9961f442e4a53a3f3accc14d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260925 19:13:26.254349 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec417a9961f442e4a53a3f3accc14d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec417a9961f442e4a53a3f3accc14d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260925 19:13:26.254379 16103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:26.254441 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:26.254801 16110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:26.255228 16110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:26.255685 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16030
I20260925 19:13:26.255813 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:26.256019 28345 external_mini_cluster.cc:1428] Setting key 1a48b5aa6c8d4afc1c6281bbe0c9cde9
I20260925 19:13:26.257136 16110 catalog_manager.cc:1347] Generated new cluster ID: bce24c1d6fb74864bc2a532ae733a7fb
I20260925 19:13:26.257193 16110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:13:26.266242 16110 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:26.266846 16110 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:26.268241 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:26.256519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55883 (local address 127.0.0.1:45377)
0925 19:13:26.256694 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:26.256703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:26.256739 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:13:26.256879 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:26.256887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:26.256953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:26.257092 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:26.257907 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:26.259283 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:26.260326 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:26.260911 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:26.261001 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:26.264214 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:26.264219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:26.264219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:26.264231 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:26.266395 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:26.266943 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:26.266946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:26.266947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:26.267060 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:26.267318 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:26.267318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:26.267319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:26.267576 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:26.267759 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:26.267909 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:13:26.268011 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260925 19:13:26.269336 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:26.279819 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:26.271891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60039 (local address 127.0.0.1:45377)
0925 19:13:26.271924 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:26.271926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:26.272071 (+ 145us) server_negotiation.cc:407] Connection header received
0925 19:13:26.272134 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:26.272135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:26.272145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:26.272196 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:26.272880 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:26.273635 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:26.274783 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:26.275094 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:26.275156 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:26.276805 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:26.276806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:26.276807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:26.276814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:26.278387 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:26.278835 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:26.278836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:26.278837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:26.278891 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:26.279137 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:26.279138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:26.279138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:26.279269 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:26.279416 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:26.279655 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 19:13:26.279724 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:13:26.280167 16110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec417a9961f442e4a53a3f3accc14d40: Generated new TSK 0
I20260925 19:13:26.280320 16110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:26.288049 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16030
W20260925 19:13:26.292191 16015 connection.cc:535] client connection to 127.0.0.1:45377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45377): Connection reset by peer (error 104)
2026-09-25T19:13:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16129](info): set up 2 sockets
Sep 25 19:13:26 dist-test-slave-fcdn krb5kdc[16129](info): commencing operation
krb5kdc: starting...
W20260925 19:13:27.327734 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:27 dist-test-slave-fcdn krb5kdc[16129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:27.447011 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:27.526611 16142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:27.526896 16142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:27.526944 16142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:27.528155 16142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:27.528225 16142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:27.528257 16142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:27.528286 16142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:27.528316 16142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:27.529803 16142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:27.530077 16142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:27.530296 16142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:27.534934 16148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:27.534952 16150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:27.534981 16147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:27.535358 16142 server_base.cc:1034] running on GCE node
I20260925 19:13:27.535670 16142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:27.535948 16142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:27.537111 16142 hybrid_clock.cc:648] HybridClock initialized: now 1790363607537096 us; error 29 us; skew 500 ppm
Sep 25 19:13:27 dist-test-slave-fcdn krb5kdc[16129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:27.539690 16142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:27.540055 16142 webserver.cc:460] Webserver started at http://127.0.0.1:34225/ using document root <none> and password file <none>
I20260925 19:13:27.540241 16142 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:27.540282 16142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:27.540374 16142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:27.541409 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "8559b2348ff9459396986e90471191d8"
format_stamp: "Formatted at 2026-09-25 19:13:27 on dist-test-slave-fcdn"
server_key: "8e366b6187de66873be2116608d5ebfa"
server_key_iv: "030b2e08c815f40491a32623a21bddef"
server_key_version: "encryptionkey@0"
I20260925 19:13:27.541699 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8559b2348ff9459396986e90471191d8"
format_stamp: "Formatted at 2026-09-25 19:13:27 on dist-test-slave-fcdn"
server_key: "8e366b6187de66873be2116608d5ebfa"
server_key_iv: "030b2e08c815f40491a32623a21bddef"
server_key_version: "encryptionkey@0"
I20260925 19:13:27.542950 16142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:27.543670 16157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:27.543876 16142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:27.543967 16142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "8559b2348ff9459396986e90471191d8"
format_stamp: "Formatted at 2026-09-25 19:13:27 on dist-test-slave-fcdn"
server_key: "8e366b6187de66873be2116608d5ebfa"
server_key_iv: "030b2e08c815f40491a32623a21bddef"
server_key_version: "encryptionkey@0"
I20260925 19:13:27.544016 16142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:27.569749 16142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:27.571756 16142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:27.571874 16142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:27.575522 16142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33793
I20260925 19:13:27.575591 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33793 every 8 connection(s)
I20260925 19:13:27.575953 16142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:27.576864 16210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:27.579203 16210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8: Bootstrap starting.
I20260925 19:13:27.579798 16210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:27.580078 16210 log.cc:824] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:27.580778 16210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8: No bootstrap required, opened a new log
I20260925 19:13:27.581511 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16142
I20260925 19:13:27.581607 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:27.581835 28345 external_mini_cluster.cc:1428] Setting key a41c414badf44cad11c83b4c22ffc1d0
I20260925 19:13:27.582981 16210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260925 19:13:27.583159 16210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:27.583199 16210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559b2348ff9459396986e90471191d8, State: Initialized, Role: FOLLOWER
I20260925 19:13:27.583341 16210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260925 19:13:27.583437 16210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:27.583470 16210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:27.583521 16210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:27.584267 16210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260925 19:13:27.584380 16210 leader_election.cc:302] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8559b2348ff9459396986e90471191d8; no voters:
I20260925 19:13:27.584533 16210 leader_election.cc:288] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:27.584597 16215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:27.584743 16215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8559b2348ff9459396986e90471191d8, State: Running, Role: LEADER
I20260925 19:13:27.584759 16210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:27.584828 16215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260925 19:13:27.585147 16215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8559b2348ff9459396986e90471191d8. Latest consensus state: current_term: 1 leader_uuid: "8559b2348ff9459396986e90471191d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260925 19:13:27.585222 16215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:27.585386 16216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8559b2348ff9459396986e90471191d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8559b2348ff9459396986e90471191d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260925 19:13:27.585460 16216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:27.585585 16223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:27.586401 16223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:27.588176 16223 catalog_manager.cc:1347] Generated new cluster ID: daacbed542e8487ebfe2ebc41369bd7f
I20260925 19:13:27.588232 16223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:27.588665 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:27.582371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49279 (local address 127.0.0.1:33793)
0925 19:13:27.582574 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:27.582583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:27.582618 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:13:27.582944 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:27.582952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:27.583016 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:27.583168 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:27.583404 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:27.585186 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:27.586397 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:27.587001 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:27.587128 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:27.587986 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:27.587990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:27.587990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:27.588237 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:27.588278 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:27.588293 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:27.588375 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:27.588469 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:13:27.588495 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260925 19:13:27.592098 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:27.598430 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:27.593444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58623 (local address 127.0.0.1:33793)
0925 19:13:27.593480 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:27.593482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:27.595763 (+ 2281us) server_negotiation.cc:407] Connection header received
0925 19:13:27.595833 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:27.595834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:27.595844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:27.595889 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:27.596010 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:27.596467 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:27.597214 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:27.597420 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:27.597504 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:27.598169 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:27.598170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:27.598171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:27.598175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:27.598188 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:27.598192 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:27.598251 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:27.598322 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:13:27.598334 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:27.610116 16223 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:27.610654 16223 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:27.615178 16223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8559b2348ff9459396986e90471191d8: Generated new TSK 0
I20260925 19:13:27.615340 16223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:27.618312 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16142
W20260925 19:13:27.622635 16127 connection.cc:535] client connection to 127.0.0.1:33793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33793): Connection reset by peer (error 104)
2026-09-25T19:13:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:27 dist-test-slave-fcdn krb5kdc[16241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:27 dist-test-slave-fcdn krb5kdc[16241](info): set up 2 sockets
Sep 25 19:13:27 dist-test-slave-fcdn krb5kdc[16241](info): commencing operation
krb5kdc: starting...
W20260925 19:13:28.655562 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:28 dist-test-slave-fcdn krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:28.772051 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:28.850817 16254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:28.851065 16254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:28.851109 16254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:28.852277 16254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:28.852344 16254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:28.852368 16254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:28.852391 16254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:28.852421 16254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:28.853896 16254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:28.854156 16254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:28.854384 16254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:28.859344 16262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:28.859558 16254 server_base.cc:1034] running on GCE node
W20260925 19:13:28.859344 16259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:28.859387 16260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:28.859917 16254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:28.860198 16254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:28.861356 16254 hybrid_clock.cc:648] HybridClock initialized: now 1790363608861343 us; error 31 us; skew 500 ppm
Sep 25 19:13:28 dist-test-slave-fcdn krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:28.863914 16254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:28.864334 16254 webserver.cc:460] Webserver started at http://127.0.0.1:35949/ using document root <none> and password file <none>
I20260925 19:13:28.864580 16254 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:28.864631 16254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:28.864787 16254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:28.865885 16254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "8b24328e569d4dc2a151cbce8cb83351"
format_stamp: "Formatted at 2026-09-25 19:13:28 on dist-test-slave-fcdn"
server_key: "cd16c65a18ef7143e92725d1a8d315ae"
server_key_iv: "364c22c7b5fddbd7865e987529006076"
server_key_version: "encryptionkey@0"
I20260925 19:13:28.866231 16254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8b24328e569d4dc2a151cbce8cb83351"
format_stamp: "Formatted at 2026-09-25 19:13:28 on dist-test-slave-fcdn"
server_key: "cd16c65a18ef7143e92725d1a8d315ae"
server_key_iv: "364c22c7b5fddbd7865e987529006076"
server_key_version: "encryptionkey@0"
I20260925 19:13:28.867520 16254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:28.868357 16269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:28.868538 16254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:28.868608 16254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "8b24328e569d4dc2a151cbce8cb83351"
format_stamp: "Formatted at 2026-09-25 19:13:28 on dist-test-slave-fcdn"
server_key: "cd16c65a18ef7143e92725d1a8d315ae"
server_key_iv: "364c22c7b5fddbd7865e987529006076"
server_key_version: "encryptionkey@0"
I20260925 19:13:28.868693 16254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:28.871946 16254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:28.873800 16254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:28.873958 16254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:28.877792 16254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36057
I20260925 19:13:28.877823 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36057 every 8 connection(s)
I20260925 19:13:28.878160 16254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:28.879113 16322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:28.881407 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351: Bootstrap starting.
I20260925 19:13:28.882000 16322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:28.882227 16322 log.cc:824] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:28.882858 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351: No bootstrap required, opened a new log
I20260925 19:13:28.884481 16322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } }
I20260925 19:13:28.884614 16322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:28.884636 16322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b24328e569d4dc2a151cbce8cb83351, State: Initialized, Role: FOLLOWER
I20260925 19:13:28.884735 16322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } }
I20260925 19:13:28.884814 16322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:28.884848 16322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:28.884922 16322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:28.885448 16322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } }
I20260925 19:13:28.885561 16322 leader_election.cc:302] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b24328e569d4dc2a151cbce8cb83351; no voters:
I20260925 19:13:28.885732 16322 leader_election.cc:288] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:28.885787 16325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:28.885974 16322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:28.886010 16325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [term 1 LEADER]: Becoming Leader. State: Replica: 8b24328e569d4dc2a151cbce8cb83351, State: Running, Role: LEADER
I20260925 19:13:28.886173 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16254
I20260925 19:13:28.886173 16325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } }
I20260925 19:13:28.886301 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:28.886482 28345 external_mini_cluster.cc:1428] Setting key e73cec7032c55b69c30d0ffb82f93f84
I20260925 19:13:28.886620 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b24328e569d4dc2a151cbce8cb83351. Latest consensus state: current_term: 1 leader_uuid: "8b24328e569d4dc2a151cbce8cb83351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } } }
I20260925 19:13:28.886703 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:28.887117 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b24328e569d4dc2a151cbce8cb83351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b24328e569d4dc2a151cbce8cb83351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36057 } } }
I20260925 19:13:28.887228 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:28.887305 16329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:28.887873 16329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:28.889251 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:28.887119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60771 (local address 127.0.0.1:36057)
0925 19:13:28.887355 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:28.887367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:28.887405 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:13:28.887542 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:28.887550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:28.887582 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:28.887754 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:28.888733 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:28.888736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:28.888736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:28.888741 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:28.888793 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:28.888804 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:28.888881 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:28.889060 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 19:13:28.889106 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":56,"threads_started":1}
I20260925 19:13:28.889518 16329 catalog_manager.cc:1347] Generated new cluster ID: bbbfb85bab9c4445b1e298761786ff2b
I20260925 19:13:28.889616 16329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:28.890120 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:28.896806 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:28.891453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57271 (local address 127.0.0.1:36057)
0925 19:13:28.891487 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:28.891489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:28.895799 (+ 4310us) server_negotiation.cc:407] Connection header received
0925 19:13:28.895887 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:28.895888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:28.895894 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:28.895954 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:28.896558 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:28.896560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:28.896560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:28.896564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:28.896578 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:28.896582 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:28.896639 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:28.896693 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:13:28.896706 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:13:28.930968 16329 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:28.931167 16329 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:28.936198 16329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b24328e569d4dc2a151cbce8cb83351: Generated new TSK 0
I20260925 19:13:28.936400 16329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:28.944842 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16254
2026-09-25T19:13:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:28 dist-test-slave-fcdn krb5kdc[16353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:28 dist-test-slave-fcdn krb5kdc[16353](info): set up 2 sockets
Sep 25 19:13:28 dist-test-slave-fcdn krb5kdc[16353](info): commencing operation
krb5kdc: starting...
W20260925 19:13:29.967660 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:30.086289 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:30.166476 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:30.166708 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:30.166738 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:30.167968 16366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:30.168009 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:30.168023 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:30.168036 16366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:30.168048 16366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:30.169554 16366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:30.169795 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:30.170043 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:30.175086 16366 server_base.cc:1034] running on GCE node
W20260925 19:13:30.175205 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:30.175208 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:30.175248 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:30.175547 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:30.175906 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:30.177053 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790363610177040 us; error 29 us; skew 500 ppm
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:30.179497 16366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:30.179997 16366 webserver.cc:460] Webserver started at http://127.0.0.1:40193/ using document root <none> and password file <none>
I20260925 19:13:30.180239 16366 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:30.180311 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:30.180481 16366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:30.181536 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "948951467efa4d4ba7e4688c855d44c7"
format_stamp: "Formatted at 2026-09-25 19:13:30 on dist-test-slave-fcdn"
server_key: "414e7c2327bfdf7fdc1bff9fae78f6f1"
server_key_iv: "ed7bcab08f449fc74c6ee724a788d145"
server_key_version: "encryptionkey@0"
I20260925 19:13:30.181908 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "948951467efa4d4ba7e4688c855d44c7"
format_stamp: "Formatted at 2026-09-25 19:13:30 on dist-test-slave-fcdn"
server_key: "414e7c2327bfdf7fdc1bff9fae78f6f1"
server_key_iv: "ed7bcab08f449fc74c6ee724a788d145"
server_key_version: "encryptionkey@0"
I20260925 19:13:30.183223 16366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:30.184067 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:30.184293 16366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:13:30.184382 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "948951467efa4d4ba7e4688c855d44c7"
format_stamp: "Formatted at 2026-09-25 19:13:30 on dist-test-slave-fcdn"
server_key: "414e7c2327bfdf7fdc1bff9fae78f6f1"
server_key_iv: "ed7bcab08f449fc74c6ee724a788d145"
server_key_version: "encryptionkey@0"
I20260925 19:13:30.184465 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:30.194165 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:30.196115 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:30.196273 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:30.200582 16366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35435
I20260925 19:13:30.200614 16433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35435 every 8 connection(s)
I20260925 19:13:30.200989 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:30.201836 16434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:30.204178 16434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7: Bootstrap starting.
I20260925 19:13:30.204756 16434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:30.205035 16434 log.cc:824] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:30.205714 16434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7: No bootstrap required, opened a new log
I20260925 19:13:30.207279 16434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } }
I20260925 19:13:30.207412 16434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:30.207433 16434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948951467efa4d4ba7e4688c855d44c7, State: Initialized, Role: FOLLOWER
I20260925 19:13:30.207581 16434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } }
I20260925 19:13:30.207706 16434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:30.207731 16434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:30.207795 16434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:30.208361 16434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } }
I20260925 19:13:30.208470 16434 leader_election.cc:302] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 948951467efa4d4ba7e4688c855d44c7; no voters:
I20260925 19:13:30.208647 16434 leader_election.cc:288] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:30.208686 16437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:30.208873 16437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [term 1 LEADER]: Becoming Leader. State: Replica: 948951467efa4d4ba7e4688c855d44c7, State: Running, Role: LEADER
I20260925 19:13:30.208894 16434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:30.208956 16437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } }
I20260925 19:13:30.209398 16438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "948951467efa4d4ba7e4688c855d44c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } } }
I20260925 19:13:30.209419 16439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948951467efa4d4ba7e4688c855d44c7. Latest consensus state: current_term: 1 leader_uuid: "948951467efa4d4ba7e4688c855d44c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948951467efa4d4ba7e4688c855d44c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35435 } } }
I20260925 19:13:30.209559 16439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:30.209741 16438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:30.209944 16441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:30.210460 16441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:30.210992 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16366
I20260925 19:13:30.211113 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:30.211292 28345 external_mini_cluster.cc:1428] Setting key 6b6456090d95f555f631d5b58452dcdb
I20260925 19:13:30.212317 16441 catalog_manager.cc:1347] Generated new cluster ID: 472ea8260cce4844aeccfcf164dc16bd
I20260925 19:13:30.212363 16441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:13:30.222812 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:30.211712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47961 (local address 127.0.0.1:35435)
0925 19:13:30.211902 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:30.211912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:30.211948 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:13:30.212140 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:30.212148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:30.212203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:30.212342 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:30.213068 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:30.214497 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:30.215532 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:30.216211 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:30.216323 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:30.219048 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:30.219053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:30.219053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:30.219065 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:30.221108 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:30.221491 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:30.221494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:30.221494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:30.221601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:30.221798 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:30.221799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:30.221799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:30.222094 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:30.222265 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:30.222332 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:13:30.222563 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260925 19:13:30.223750 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:30.226673 16441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:30.227216 16441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:30.230856 16441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 948951467efa4d4ba7e4688c855d44c7: Generated new TSK 0
I20260925 19:13:30.231029 16441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:30.232486 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:30.225193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51059 (local address 127.0.0.1:35435)
0925 19:13:30.225230 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:30.225231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:30.225389 (+ 158us) server_negotiation.cc:407] Connection header received
0925 19:13:30.225435 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:30.225435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:30.225460 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:30.225515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:30.226123 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:30.226834 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:30.227869 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:30.228181 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:30.228250 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:30.229568 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:30.229569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:30.229569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:30.229576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:30.231278 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:30.231737 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:30.231738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:30.231738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:30.231793 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:30.231997 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:30.231998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:30.231998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:30.232094 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:30.232167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:30.232325 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:13:30.232394 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:13:30.234280 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16366
2026-09-25T19:13:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16465](info): set up 2 sockets
Sep 25 19:13:30 dist-test-slave-fcdn krb5kdc[16465](info): commencing operation
krb5kdc: starting...
W20260925 19:13:31.263669 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:31 dist-test-slave-fcdn krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:31.380488 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:31.459254 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:31.459455 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:31.459476 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:31.460654 16478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:31.460716 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:31.460729 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:31.460741 16478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:31.460753 16478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:31.462193 16478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:31.462440 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:31.462670 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:31.467687 16478 server_base.cc:1034] running on GCE node
W20260925 19:13:31.467658 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:31.467653 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:31.467705 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:31.468164 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:31.468448 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:31.469592 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790363611469567 us; error 63 us; skew 500 ppm
Sep 25 19:13:31 dist-test-slave-fcdn krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:31.477523 16478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:31.477985 16478 webserver.cc:460] Webserver started at http://127.0.0.1:40301/ using document root <none> and password file <none>
I20260925 19:13:31.478168 16478 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:31.478207 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:31.478291 16478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:31.479271 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "f418b23682934da1b66a9edf65f7796c"
format_stamp: "Formatted at 2026-09-25 19:13:31 on dist-test-slave-fcdn"
server_key: "8f113c49168da03c011ca55b293090a3"
server_key_iv: "599702e87bf025dde5aac1fcf7a83296"
server_key_version: "encryptionkey@0"
I20260925 19:13:31.479584 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "f418b23682934da1b66a9edf65f7796c"
format_stamp: "Formatted at 2026-09-25 19:13:31 on dist-test-slave-fcdn"
server_key: "8f113c49168da03c011ca55b293090a3"
server_key_iv: "599702e87bf025dde5aac1fcf7a83296"
server_key_version: "encryptionkey@0"
I20260925 19:13:31.480897 16478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:31.481600 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:31.481775 16478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:31.481873 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "f418b23682934da1b66a9edf65f7796c"
format_stamp: "Formatted at 2026-09-25 19:13:31 on dist-test-slave-fcdn"
server_key: "8f113c49168da03c011ca55b293090a3"
server_key_iv: "599702e87bf025dde5aac1fcf7a83296"
server_key_version: "encryptionkey@0"
I20260925 19:13:31.481953 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:31.509814 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:31.511806 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:31.511981 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:31.515962 16478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44095
I20260925 19:13:31.516019 16545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44095 every 8 connection(s)
I20260925 19:13:31.516392 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:31.517266 16546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:31.519431 16546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c: Bootstrap starting.
I20260925 19:13:31.520041 16546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:31.520273 16546 log.cc:824] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:31.520931 16546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c: No bootstrap required, opened a new log
I20260925 19:13:31.522420 16546 raft_consensus.cc:348] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } }
I20260925 19:13:31.522554 16546 raft_consensus.cc:374] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:31.522575 16546 raft_consensus.cc:729] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f418b23682934da1b66a9edf65f7796c, State: Initialized, Role: FOLLOWER
I20260925 19:13:31.522675 16546 consensus_queue.cc:260] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } }
I20260925 19:13:31.522733 16546 raft_consensus.cc:388] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:31.522753 16546 raft_consensus.cc:482] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:31.522784 16546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:31.523244 16546 raft_consensus.cc:504] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } }
I20260925 19:13:31.523324 16546 leader_election.cc:302] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f418b23682934da1b66a9edf65f7796c; no voters:
I20260925 19:13:31.523447 16546 leader_election.cc:288] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:31.523518 16549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:31.523703 16549 raft_consensus.cc:686] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [term 1 LEADER]: Becoming Leader. State: Replica: f418b23682934da1b66a9edf65f7796c, State: Running, Role: LEADER
I20260925 19:13:31.523738 16546 sys_catalog.cc:564] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:31.523790 16549 consensus_queue.cc:237] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } }
I20260925 19:13:31.524204 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f418b23682934da1b66a9edf65f7796c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } } }
I20260925 19:13:31.524235 16551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f418b23682934da1b66a9edf65f7796c. Latest consensus state: current_term: 1 leader_uuid: "f418b23682934da1b66a9edf65f7796c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f418b23682934da1b66a9edf65f7796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44095 } } }
I20260925 19:13:31.524374 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:31.524400 16551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:31.524811 16553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:31.525157 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16478
I20260925 19:13:31.525275 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:31.525342 16553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:31.525457 28345 external_mini_cluster.cc:1428] Setting key a53b16633ca78a162b368f71031aba89
I20260925 19:13:31.526940 16553 catalog_manager.cc:1347] Generated new cluster ID: 88274616de1e4fc4aca79e867445423b
I20260925 19:13:31.526993 16553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:31.530779 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:31.526077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34369 (local address 127.0.0.1:44095)
0925 19:13:31.526266 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:31.526276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:31.526473 (+ 197us) server_negotiation.cc:407] Connection header received
0925 19:13:31.526621 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:31.526628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:31.526687 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:31.527080 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:31.527162 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:31.528116 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:31.528922 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:31.529329 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:31.529447 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:31.530149 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:31.530153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:31.530153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:31.530328 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:31.530360 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:31.530371 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:31.530447 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:31.530511 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:13:31.530571 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260925 19:13:31.531668 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:31.538358 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:31.532978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36347 (local address 127.0.0.1:44095)
0925 19:13:31.533014 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:31.533016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:31.535764 (+ 2748us) server_negotiation.cc:407] Connection header received
0925 19:13:31.535827 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:31.535828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:31.535839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:31.535895 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:31.536015 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:31.536476 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:31.537181 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:31.537393 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:31.537469 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:31.538118 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:31.538119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:31.538119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:31.538124 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:31.538138 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:31.538142 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:31.538195 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:31.538253 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:13:31.538275 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:13:31.550089 16553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:31.550614 16553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:31.553157 16553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f418b23682934da1b66a9edf65f7796c: Generated new TSK 0
I20260925 19:13:31.553335 16553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:31.557942 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16478
2026-09-25T19:13:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:31 dist-test-slave-fcdn krb5kdc[16577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:31 dist-test-slave-fcdn krb5kdc[16577](info): set up 2 sockets
Sep 25 19:13:31 dist-test-slave-fcdn krb5kdc[16577](info): commencing operation
krb5kdc: starting...
W20260925 19:13:32.579658 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:32 dist-test-slave-fcdn krb5kdc[16577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:32.695315 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:32.774955 16590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:32.775178 16590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:32.775215 16590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:32.776428 16590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:32.776494 16590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:32.776518 16590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:32.776552 16590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:32.776575 16590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:32.777992 16590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:32.778254 16590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:32.778474 16590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:32.783262 16595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:32.783273 16598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:32.783385 16590 server_base.cc:1034] running on GCE node
W20260925 19:13:32.783273 16596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:32.783764 16590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:32.784087 16590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:32.785257 16590 hybrid_clock.cc:648] HybridClock initialized: now 1790363612785242 us; error 27 us; skew 500 ppm
Sep 25 19:13:32 dist-test-slave-fcdn krb5kdc[16577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:32.787765 16590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:32.788180 16590 webserver.cc:460] Webserver started at http://127.0.0.1:35791/ using document root <none> and password file <none>
I20260925 19:13:32.788407 16590 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:32.788471 16590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:32.788602 16590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:32.789644 16590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "7e3cd66313ad46858c528a5ad9983f54"
format_stamp: "Formatted at 2026-09-25 19:13:32 on dist-test-slave-fcdn"
server_key: "d9462b4e500dd1ae12f721e42db945d4"
server_key_iv: "cb96aff8ff45e2f9b83590de6aa3a5d0"
server_key_version: "encryptionkey@0"
I20260925 19:13:32.790004 16590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "7e3cd66313ad46858c528a5ad9983f54"
format_stamp: "Formatted at 2026-09-25 19:13:32 on dist-test-slave-fcdn"
server_key: "d9462b4e500dd1ae12f721e42db945d4"
server_key_iv: "cb96aff8ff45e2f9b83590de6aa3a5d0"
server_key_version: "encryptionkey@0"
I20260925 19:13:32.791409 16590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:32.792166 16605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:32.792341 16590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:32.792433 16590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "7e3cd66313ad46858c528a5ad9983f54"
format_stamp: "Formatted at 2026-09-25 19:13:32 on dist-test-slave-fcdn"
server_key: "d9462b4e500dd1ae12f721e42db945d4"
server_key_iv: "cb96aff8ff45e2f9b83590de6aa3a5d0"
server_key_version: "encryptionkey@0"
I20260925 19:13:32.792513 16590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:32.795656 16590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:32.797876 16590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:32.798024 16590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:32.801837 16590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42985
I20260925 19:13:32.801892 16657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42985 every 8 connection(s)
I20260925 19:13:32.802193 16590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:32.803140 16658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:32.805358 16658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54: Bootstrap starting.
I20260925 19:13:32.805917 16658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:32.806195 16658 log.cc:824] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:32.806838 16658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54: No bootstrap required, opened a new log
I20260925 19:13:32.808490 16658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260925 19:13:32.808622 16658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:32.808645 16658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3cd66313ad46858c528a5ad9983f54, State: Initialized, Role: FOLLOWER
I20260925 19:13:32.808821 16658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260925 19:13:32.808905 16658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:32.808955 16658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:32.809005 16658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:32.809518 16658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260925 19:13:32.809599 16658 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3cd66313ad46858c528a5ad9983f54; no voters:
I20260925 19:13:32.809758 16658 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:32.809828 16661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:32.809877 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16590
I20260925 19:13:32.809991 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:32.810009 16661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [term 1 LEADER]: Becoming Leader. State: Replica: 7e3cd66313ad46858c528a5ad9983f54, State: Running, Role: LEADER
I20260925 19:13:32.810015 16658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:32.810099 16661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260925 19:13:32.810179 28345 external_mini_cluster.cc:1428] Setting key f36c01647a27fb8438dd0bce07936ffe
I20260925 19:13:32.810403 16661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3cd66313ad46858c528a5ad9983f54. Latest consensus state: current_term: 1 leader_uuid: "7e3cd66313ad46858c528a5ad9983f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } } }
I20260925 19:13:32.810660 16661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:32.811224 16667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:32.811213 16662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3cd66313ad46858c528a5ad9983f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3cd66313ad46858c528a5ad9983f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } } }
I20260925 19:13:32.811431 16662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:32.811849 16667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:32.813473 16667 catalog_manager.cc:1347] Generated new cluster ID: 9b0ba444b7b943a782e0cbf9e9b2827a
I20260925 19:13:32.813540 16667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:32.813683 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:32.810836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53045 (local address 127.0.0.1:42985)
0925 19:13:32.811129 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:32.811141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:32.811463 (+ 322us) server_negotiation.cc:407] Connection header received
0925 19:13:32.811771 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:32.811779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:32.811811 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:32.811918 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:32.812997 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:32.813002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:32.813002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:32.813007 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:32.813052 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:32.813066 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:32.813168 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:32.813250 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 19:13:32.813293 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
I20260925 19:13:32.814666 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:32.817279 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:32.816248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49931 (local address 127.0.0.1:42985)
0925 19:13:32.816277 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:32.816278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:32.816396 (+ 118us) server_negotiation.cc:407] Connection header received
0925 19:13:32.816445 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:32.816446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:32.816451 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:32.816498 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:32.817077 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:32.817078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:32.817078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:32.817082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:32.817095 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:32.817099 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:32.817150 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:32.817212 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:13:32.817224 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:13:32.830802 16667 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:32.830984 16667 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:32.838363 16667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3cd66313ad46858c528a5ad9983f54: Generated new TSK 0
I20260925 19:13:32.838547 16667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:32.843209 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16590
2026-09-25T19:13:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:32 dist-test-slave-fcdn krb5kdc[16689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:32 dist-test-slave-fcdn krb5kdc[16689](info): set up 2 sockets
Sep 25 19:13:32 dist-test-slave-fcdn krb5kdc[16689](info): commencing operation
krb5kdc: starting...
W20260925 19:13:33.875809 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:33 dist-test-slave-fcdn krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:33.992215 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:34.074863 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:34.075096 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:34.075131 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:34.076349 16702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:34.076413 16702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:34.076436 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:34.076458 16702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:34.076488 16702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:34.077975 16702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:34.078244 16702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:34.078469 16702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:34.083181 16707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:34.083324 16702 server_base.cc:1034] running on GCE node
W20260925 19:13:34.083240 16708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:34.083168 16710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:34.083717 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:34.084051 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:34.085214 16702 hybrid_clock.cc:648] HybridClock initialized: now 1790363614085201 us; error 25 us; skew 500 ppm
Sep 25 19:13:34 dist-test-slave-fcdn krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:34.087586 16702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:34.088169 16702 webserver.cc:460] Webserver started at http://127.0.0.1:35549/ using document root <none> and password file <none>
I20260925 19:13:34.088421 16702 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:34.088487 16702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:34.088618 16702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:34.089679 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "be65a985a7be436687c8ab5abf167e31"
format_stamp: "Formatted at 2026-09-25 19:13:34 on dist-test-slave-fcdn"
server_key: "aeeb5ae2f4f0f198b79bcd046cb60c5b"
server_key_iv: "9f946a9251a6fd6b53a77548fadacf44"
server_key_version: "encryptionkey@0"
I20260925 19:13:34.090044 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "be65a985a7be436687c8ab5abf167e31"
format_stamp: "Formatted at 2026-09-25 19:13:34 on dist-test-slave-fcdn"
server_key: "aeeb5ae2f4f0f198b79bcd046cb60c5b"
server_key_iv: "9f946a9251a6fd6b53a77548fadacf44"
server_key_version: "encryptionkey@0"
I20260925 19:13:34.091359 16702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:34.092221 16717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:34.092417 16702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:13:34.092511 16702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "be65a985a7be436687c8ab5abf167e31"
format_stamp: "Formatted at 2026-09-25 19:13:34 on dist-test-slave-fcdn"
server_key: "aeeb5ae2f4f0f198b79bcd046cb60c5b"
server_key_iv: "9f946a9251a6fd6b53a77548fadacf44"
server_key_version: "encryptionkey@0"
I20260925 19:13:34.092597 16702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:34.105101 16702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:34.106956 16702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:34.107116 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:34.111192 16702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46743
I20260925 19:13:34.111228 16769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46743 every 8 connection(s)
I20260925 19:13:34.111650 16702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:34.112505 16770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:34.114706 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31: Bootstrap starting.
I20260925 19:13:34.115234 16770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:34.115460 16770 log.cc:824] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:34.116128 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31: No bootstrap required, opened a new log
I20260925 19:13:34.116662 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16702
I20260925 19:13:34.116778 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:34.116971 28345 external_mini_cluster.cc:1428] Setting key 84c170c8dedadbb29db1e72e469c2671
I20260925 19:13:34.118252 16770 raft_consensus.cc:348] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260925 19:13:34.118420 16770 raft_consensus.cc:374] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:34.118460 16770 raft_consensus.cc:729] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be65a985a7be436687c8ab5abf167e31, State: Initialized, Role: FOLLOWER
I20260925 19:13:34.118652 16770 consensus_queue.cc:260] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260925 19:13:34.118757 16770 raft_consensus.cc:388] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:34.118790 16770 raft_consensus.cc:482] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:34.118849 16770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:34.119870 16770 raft_consensus.cc:504] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260925 19:13:34.119958 16770 leader_election.cc:302] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be65a985a7be436687c8ab5abf167e31; no voters:
I20260925 19:13:34.120172 16770 leader_election.cc:288] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:34.120190 16775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:34.120301 16775 raft_consensus.cc:686] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [term 1 LEADER]: Becoming Leader. State: Replica: be65a985a7be436687c8ab5abf167e31, State: Running, Role: LEADER
I20260925 19:13:34.120432 16770 sys_catalog.cc:564] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:34.120432 16775 consensus_queue.cc:237] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260925 19:13:34.120877 16777 sys_catalog.cc:455] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be65a985a7be436687c8ab5abf167e31. Latest consensus state: current_term: 1 leader_uuid: "be65a985a7be436687c8ab5abf167e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } } }
I20260925 19:13:34.121001 16777 sys_catalog.cc:458] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:34.121178 16776 sys_catalog.cc:455] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be65a985a7be436687c8ab5abf167e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be65a985a7be436687c8ab5abf167e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } } }
I20260925 19:13:34.121272 16776 sys_catalog.cc:458] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:34.121557 16782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:34.122062 16782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:34.124013 16782 catalog_manager.cc:1347] Generated new cluster ID: 5920728d70d44e91862e90336ccb07c1
I20260925 19:13:34.124065 16782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:13:34 dist-test-slave-fcdn krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:13:34.128983 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:34.117480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:40691 (local address 127.0.0.1:46743)
0925 19:13:34.117666 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:34.117675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:34.117703 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:13:34.118227 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:34.118233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:34.118281 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:34.118401 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:34.119204 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:34.120330 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:34.121442 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:34.122094 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:34.122522 (+ 428us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:34.125074 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:34.125078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:34.125078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:34.125091 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:34.127124 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:34.127558 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:34.127561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:34.127561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:34.127720 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:34.127964 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:34.127965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:34.127965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:34.128280 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:34.128467 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:34.128554 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 19:13:34.128776 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260925 19:13:34.130085 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:34.138840 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:34.131502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:38441 (local address 127.0.0.1:46743)
0925 19:13:34.131537 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:34.131539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:34.131652 (+ 113us) server_negotiation.cc:407] Connection header received
0925 19:13:34.131713 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:34.131714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:34.131723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:34.131776 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:34.132401 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:34.133134 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:34.134152 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:34.134468 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:34.134526 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:34.135925 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:34.135926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:34.135927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:34.135933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:34.137511 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:34.137924 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:34.137925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:34.137925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:34.137978 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:34.138226 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:34.138227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:34.138228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:34.138368 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:34.138445 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:34.138680 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 19:13:34.138746 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:34.142138 16782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:34.142735 16782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:34.148900 16782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be65a985a7be436687c8ab5abf167e31: Generated new TSK 0
I20260925 19:13:34.149116 16782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:34.155282 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16702
W20260925 19:13:34.159526 16687 connection.cc:535] client connection to 127.0.0.1:46743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46743): Connection reset by peer (error 104)
2026-09-25T19:13:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:34 dist-test-slave-fcdn krb5kdc[16801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:34 dist-test-slave-fcdn krb5kdc[16801](info): set up 2 sockets
Sep 25 19:13:34 dist-test-slave-fcdn krb5kdc[16801](info): commencing operation
krb5kdc: starting...
W20260925 19:13:35.183759 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:35 dist-test-slave-fcdn krb5kdc[16801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:35.302521 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:35.381377 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:35.381610 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:35.381647 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:35.382787 16814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:35.382854 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:35.382877 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:35.382898 16814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:35.382928 16814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:35.384378 16814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:35.384641 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:35.384861 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:13:35.389817 16814 server_base.cc:1034] running on GCE node
W20260925 19:13:35.389798 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:35.389796 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:35.389797 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:35.390247 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:35.390511 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:35.391666 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790363615391655 us; error 29 us; skew 500 ppm
Sep 25 19:13:35 dist-test-slave-fcdn krb5kdc[16801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:35.394100 16814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:35.394531 16814 webserver.cc:460] Webserver started at http://127.0.0.1:36031/ using document root <none> and password file <none>
I20260925 19:13:35.394763 16814 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:35.394832 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:35.394969 16814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:35.396078 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "fd7fe32da63f40bfb61a9d693261a48f"
format_stamp: "Formatted at 2026-09-25 19:13:35 on dist-test-slave-fcdn"
server_key: "b6a22027461c9fde983c53e7dd9843fc"
server_key_iv: "50b6e39278e76ca9a098471155589963"
server_key_version: "encryptionkey@0"
I20260925 19:13:35.396432 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "fd7fe32da63f40bfb61a9d693261a48f"
format_stamp: "Formatted at 2026-09-25 19:13:35 on dist-test-slave-fcdn"
server_key: "b6a22027461c9fde983c53e7dd9843fc"
server_key_iv: "50b6e39278e76ca9a098471155589963"
server_key_version: "encryptionkey@0"
I20260925 19:13:35.397887 16814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:35.398597 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:35.398749 16814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:35.398854 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "fd7fe32da63f40bfb61a9d693261a48f"
format_stamp: "Formatted at 2026-09-25 19:13:35 on dist-test-slave-fcdn"
server_key: "b6a22027461c9fde983c53e7dd9843fc"
server_key_iv: "50b6e39278e76ca9a098471155589963"
server_key_version: "encryptionkey@0"
I20260925 19:13:35.398924 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:35.411216 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:35.413160 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:35.413318 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:35.417177 16814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43483
I20260925 19:13:35.417229 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43483 every 8 connection(s)
I20260925 19:13:35.417578 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:35.418395 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:35.420627 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f: Bootstrap starting.
I20260925 19:13:35.421136 16882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:35.421366 16882 log.cc:824] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:35.422019 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f: No bootstrap required, opened a new log
I20260925 19:13:35.423523 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260925 19:13:35.423727 16882 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:35.423753 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7fe32da63f40bfb61a9d693261a48f, State: Initialized, Role: FOLLOWER
I20260925 19:13:35.423861 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260925 19:13:35.423911 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:35.423933 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:35.423959 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:35.424427 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260925 19:13:35.424506 16882 leader_election.cc:302] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd7fe32da63f40bfb61a9d693261a48f; no voters:
I20260925 19:13:35.424628 16882 leader_election.cc:288] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:35.424700 16885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:35.424839 16885 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [term 1 LEADER]: Becoming Leader. State: Replica: fd7fe32da63f40bfb61a9d693261a48f, State: Running, Role: LEADER
I20260925 19:13:35.424901 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:35.424924 16885 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260925 19:13:35.425356 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7fe32da63f40bfb61a9d693261a48f. Latest consensus state: current_term: 1 leader_uuid: "fd7fe32da63f40bfb61a9d693261a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
I20260925 19:13:35.425438 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:35.425482 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd7fe32da63f40bfb61a9d693261a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7fe32da63f40bfb61a9d693261a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
I20260925 19:13:35.425541 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:35.425901 16892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:35.427135 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16814
I20260925 19:13:35.427248 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:35.427450 28345 external_mini_cluster.cc:1428] Setting key 9c880a0d6c36b5f4b21679cdf7b269d6
I20260925 19:13:35.427462 16892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:35.429019 16892 catalog_manager.cc:1347] Generated new cluster ID: 284ff09a253c46d0b5bfc0fe7e644ace
I20260925 19:13:35.429071 16892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:35.433877 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:35.428114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:37583 (local address 127.0.0.1:43483)
0925 19:13:35.428302 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:35.428310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:35.428340 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:13:35.428454 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:35.428460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:35.428550 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:35.428746 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:35.429030 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:35.430491 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:35.431555 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:35.432258 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:35.432369 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:35.433113 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:35.433118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:35.433118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:35.433326 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:35.433363 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:35.433378 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:35.433476 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:35.433577 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:13:35.433620 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 19:13:35.434569 16892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:35.434873 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:35.435179 16892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:35.439764 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:35.436407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:59039 (local address 127.0.0.1:43483)
0925 19:13:35.436440 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:35.436442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:35.436450 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:35.436505 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:35.436506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:35.436515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:35.436577 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:35.436692 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:35.437451 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:35.438441 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:35.438782 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:35.438852 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:35.439497 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:35.439498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:35.439499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:35.439503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:35.439518 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:35.439522 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:35.439583 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:35.439677 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:13:35.439689 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:35.442828 16892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd7fe32da63f40bfb61a9d693261a48f: Generated new TSK 0
I20260925 19:13:35.442982 16892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:35.447055 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16814
W20260925 19:13:35.451051 16799 connection.cc:535] client connection to 127.0.0.1:43483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43483): Connection reset by peer (error 104)
2026-09-25T19:13:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:35 dist-test-slave-fcdn krb5kdc[16913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:35 dist-test-slave-fcdn krb5kdc[16913](info): set up 2 sockets
Sep 25 19:13:35 dist-test-slave-fcdn krb5kdc[16913](info): commencing operation
krb5kdc: starting...
W20260925 19:13:36.487726 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:36 dist-test-slave-fcdn krb5kdc[16913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:36.605976 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:36.685323 16926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:36.685532 16926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:36.685555 16926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:36.686751 16926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:36.686795 16926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:36.686810 16926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:36.686823 16926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:36.686836 16926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:36.688347 16926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:36.688589 16926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:36.688848 16926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:36.693811 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:36.693843 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:36.693890 16926 server_base.cc:1034] running on GCE node
W20260925 19:13:36.693799 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:36.694235 16926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:36.694550 16926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:36.695724 16926 hybrid_clock.cc:648] HybridClock initialized: now 1790363616695713 us; error 50 us; skew 500 ppm
Sep 25 19:13:36 dist-test-slave-fcdn krb5kdc[16913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:36.702948 16926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:36.703454 16926 webserver.cc:460] Webserver started at http://127.0.0.1:44827/ using document root <none> and password file <none>
I20260925 19:13:36.703755 16926 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:36.703825 16926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:36.703984 16926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:36.705087 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "28faca740f33438c9f40fa417500948a"
format_stamp: "Formatted at 2026-09-25 19:13:36 on dist-test-slave-fcdn"
server_key: "9011c0e30b52ead33cfa2159b4850a72"
server_key_iv: "0e8902fbecf30cafd41e1edc78c11f97"
server_key_version: "encryptionkey@0"
I20260925 19:13:36.705459 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "28faca740f33438c9f40fa417500948a"
format_stamp: "Formatted at 2026-09-25 19:13:36 on dist-test-slave-fcdn"
server_key: "9011c0e30b52ead33cfa2159b4850a72"
server_key_iv: "0e8902fbecf30cafd41e1edc78c11f97"
server_key_version: "encryptionkey@0"
I20260925 19:13:36.707168 16926 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 19:13:36.709021 16941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:36.709198 16926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:13:36.709307 16926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "28faca740f33438c9f40fa417500948a"
format_stamp: "Formatted at 2026-09-25 19:13:36 on dist-test-slave-fcdn"
server_key: "9011c0e30b52ead33cfa2159b4850a72"
server_key_iv: "0e8902fbecf30cafd41e1edc78c11f97"
server_key_version: "encryptionkey@0"
I20260925 19:13:36.709396 16926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:36.712778 16926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:36.714604 16926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:36.714785 16926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:36.718957 16926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33449
I20260925 19:13:36.719007 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33449 every 8 connection(s)
I20260925 19:13:36.719348 16926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:36.720381 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:36.720407 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 16926
I20260925 19:13:36.720530 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:36.720721 28345 external_mini_cluster.cc:1428] Setting key ba3beac92178c0f916d00b739eaf2058
I20260925 19:13:36.723238 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:36.721251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:57427 (local address 127.0.0.1:33449)
0925 19:13:36.721478 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:36.721487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:36.721518 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:13:36.721989 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:36.721998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:36.722032 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:36.722183 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:36.722840 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:36.722844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:36.722844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:36.722848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:36.722891 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:36.722903 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:36.722986 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:36.723020 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 19:13:36.723063 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260925 19:13:36.723865 16994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: Bootstrap starting.
I20260925 19:13:36.724287 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:36.724632 16994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:36.724947 16994 log.cc:824] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:36.725744 16994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: No bootstrap required, opened a new log
I20260925 19:13:36.726817 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:36.725868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:37315 (local address 127.0.0.1:33449)
0925 19:13:36.725910 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:36.725912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:36.726017 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:13:36.726093 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:36.726094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:36.726098 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:36.726142 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:36.726648 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:36.726649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:36.726650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:36.726653 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:36.726665 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:36.726668 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:36.726710 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:36.726766 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 19:13:36.726777 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:13:36.728204 16994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } }
I20260925 19:13:36.728338 16994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:36.728359 16994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28faca740f33438c9f40fa417500948a, State: Initialized, Role: FOLLOWER
I20260925 19:13:36.728505 16994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } }
I20260925 19:13:36.728585 16994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:36.728627 16994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:36.728677 16994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:36.729260 16994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } }
I20260925 19:13:36.729373 16994 leader_election.cc:302] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28faca740f33438c9f40fa417500948a; no voters:
I20260925 19:13:36.729569 16994 leader_election.cc:288] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:36.729638 17004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:36.729815 17004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [term 1 LEADER]: Becoming Leader. State: Replica: 28faca740f33438c9f40fa417500948a, State: Running, Role: LEADER
I20260925 19:13:36.729821 16994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:36.729913 17004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } }
I20260925 19:13:36.730265 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28faca740f33438c9f40fa417500948a. Latest consensus state: current_term: 1 leader_uuid: "28faca740f33438c9f40fa417500948a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } } }
I20260925 19:13:36.730257 17005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28faca740f33438c9f40fa417500948a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28faca740f33438c9f40fa417500948a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33449 } } }
I20260925 19:13:36.730361 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:36.730370 17005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a [sys.catalog]: This master's current role is: LEADER
W20260925 19:13:36.731592 17020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:36.731681 17020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:36.731736 17012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:36.731933 17012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:36.733253 17012 catalog_manager.cc:1347] Generated new cluster ID: 47d23bb2284145ec9ede0485e19ba349
I20260925 19:13:36.733307 17012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:36.745347 17012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:36.745512 17012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:36.751128 17012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28faca740f33438c9f40fa417500948a: Generated new TSK 0
I20260925 19:13:36.751309 17012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:36.753922 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 16926
2026-09-25T19:13:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:36 dist-test-slave-fcdn krb5kdc[17025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:36 dist-test-slave-fcdn krb5kdc[17025](info): set up 2 sockets
Sep 25 19:13:36 dist-test-slave-fcdn krb5kdc[17025](info): commencing operation
krb5kdc: starting...
W20260925 19:13:37.788628 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:37 dist-test-slave-fcdn krb5kdc[17025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:37.906780 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:37.985585 17038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:37.985783 17038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:37.985805 17038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:37.986879 17038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:37.986914 17038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:37.986928 17038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:37.986940 17038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:37.986953 17038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:37.988417 17038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:37.988605 17038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:37.988775 17038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:37.993423 17043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:37.993579 17038 server_base.cc:1034] running on GCE node
W20260925 19:13:37.993477 17044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:37.993422 17046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:37.993921 17038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:37.994216 17038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:37.995385 17038 hybrid_clock.cc:648] HybridClock initialized: now 1790363617995367 us; error 51 us; skew 500 ppm
Sep 25 19:13:37 dist-test-slave-fcdn krb5kdc[17025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:37.997884 17038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:37.998303 17038 webserver.cc:460] Webserver started at http://127.0.0.1:40069/ using document root <none> and password file <none>
I20260925 19:13:37.998544 17038 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:37.998618 17038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:37.998772 17038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:37.999898 17038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "f1d9744841c045c0be46336976dc12fe"
format_stamp: "Formatted at 2026-09-25 19:13:37 on dist-test-slave-fcdn"
server_key: "30188b05289a2ec0138f776326588035"
server_key_iv: "1677b7bf6d2715035e89130a640b207c"
server_key_version: "encryptionkey@0"
I20260925 19:13:38.000249 17038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "f1d9744841c045c0be46336976dc12fe"
format_stamp: "Formatted at 2026-09-25 19:13:37 on dist-test-slave-fcdn"
server_key: "30188b05289a2ec0138f776326588035"
server_key_iv: "1677b7bf6d2715035e89130a640b207c"
server_key_version: "encryptionkey@0"
I20260925 19:13:38.001550 17038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:38.002265 17053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:38.002435 17038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:38.002537 17038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "f1d9744841c045c0be46336976dc12fe"
format_stamp: "Formatted at 2026-09-25 19:13:37 on dist-test-slave-fcdn"
server_key: "30188b05289a2ec0138f776326588035"
server_key_iv: "1677b7bf6d2715035e89130a640b207c"
server_key_version: "encryptionkey@0"
I20260925 19:13:38.002614 17038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:38.022560 17038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:38.024506 17038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:38.024657 17038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:38.028476 17038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39015
I20260925 19:13:38.028513 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39015 every 8 connection(s)
I20260925 19:13:38.028867 17038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:38.029714 17106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:38.031167 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17038
I20260925 19:13:38.031301 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:38.031507 28345 external_mini_cluster.cc:1428] Setting key 1a32a12f02b004ea39a55d490c72aa1f
I20260925 19:13:38.032315 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe: Bootstrap starting.
I20260925 19:13:38.032932 17106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:38.033267 17106 log.cc:824] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:38.033982 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe: No bootstrap required, opened a new log
I20260925 19:13:38.036120 17106 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } }
I20260925 19:13:38.036324 17106 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:38.036363 17106 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d9744841c045c0be46336976dc12fe, State: Initialized, Role: FOLLOWER
I20260925 19:13:38.036846 17106 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } }
I20260925 19:13:38.036953 17106 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:38.036993 17106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:38.037045 17106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:38.037802 17106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } }
I20260925 19:13:38.037920 17106 leader_election.cc:302] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1d9744841c045c0be46336976dc12fe; no voters:
I20260925 19:13:38.038100 17106 leader_election.cc:288] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:38.038117 17111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:38.038231 17111 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [term 1 LEADER]: Becoming Leader. State: Replica: f1d9744841c045c0be46336976dc12fe, State: Running, Role: LEADER
I20260925 19:13:38.038364 17111 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } }
Sep 25 19:13:38 dist-test-slave-fcdn krb5kdc[17025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:13:38.038359 17106 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:38.039903 17113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1d9744841c045c0be46336976dc12fe. Latest consensus state: current_term: 1 leader_uuid: "f1d9744841c045c0be46336976dc12fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } } }
I20260925 19:13:38.039987 17113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:38.040232 17112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1d9744841c045c0be46336976dc12fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9744841c045c0be46336976dc12fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39015 } } }
I20260925 19:13:38.040237 17127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:38.040357 17112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:38.040809 17127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:38.042310 17127 catalog_manager.cc:1347] Generated new cluster ID: c660a8afde0c4a77bd3cbc26ca4bf143
I20260925 19:13:38.042361 17127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:38.042816 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:38.032026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:46099 (local address 127.0.0.1:39015)
0925 19:13:38.032222 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:38.032230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:38.032254 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:13:38.032708 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:38.032715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:38.032765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:38.032878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:38.033688 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:38.034618 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:38.035715 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:38.036093 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:38.036183 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:38.039069 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:38.039073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:38.039073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:38.039086 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:38.041024 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:38.041406 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:38.041409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:38.041409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:38.041505 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:38.041765 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:38.041766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:38.041766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:38.042051 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:38.042218 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:38.042606 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 19:13:38.042684 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260925 19:13:38.043655 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:38.055848 17127 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:38.056695 17127 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:38.058373 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:38.051940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:36547 (local address 127.0.0.1:39015)
0925 19:13:38.051981 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:38.051982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:38.052169 (+ 187us) server_negotiation.cc:407] Connection header received
0925 19:13:38.052243 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:38.052243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:38.052254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:38.052309 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:38.052800 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:38.053275 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:38.053968 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:38.054198 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:38.054275 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:38.055422 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:38.055423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:38.055423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:38.055430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:38.057117 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:38.057544 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:38.057546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:38.057546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:38.057605 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:38.057876 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:38.057877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:38.057877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:38.057999 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:38.058102 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:38.058224 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:13:38.058284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:13:38.060086 17127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1d9744841c045c0be46336976dc12fe: Generated new TSK 0
I20260925 19:13:38.060274 17127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:38.066051 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17038
W20260925 19:13:38.069813 17023 connection.cc:535] client connection to 127.0.0.1:39015 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39015): Connection reset by peer (error 104)
2026-09-25T19:13:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:38 dist-test-slave-fcdn krb5kdc[17137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:38 dist-test-slave-fcdn krb5kdc[17137](info): set up 2 sockets
Sep 25 19:13:38 dist-test-slave-fcdn krb5kdc[17137](info): commencing operation
krb5kdc: starting...
W20260925 19:13:39.103744 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:39 dist-test-slave-fcdn krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:39.220880 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:39.299444 17150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:39.299710 17150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:39.299747 17150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:39.300905 17150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:39.300977 17150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:39.301002 17150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:39.301025 17150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:39.301054 17150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:39.302531 17150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:39.302806 17150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:39.303037 17150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:39.308009 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:39.308010 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:39.308060 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:39.308161 17150 server_base.cc:1034] running on GCE node
I20260925 19:13:39.308483 17150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:39.308773 17150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:39.309954 17150 hybrid_clock.cc:648] HybridClock initialized: now 1790363619309943 us; error 30 us; skew 500 ppm
Sep 25 19:13:39 dist-test-slave-fcdn krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:39.312424 17150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:39.312865 17150 webserver.cc:460] Webserver started at http://127.0.0.1:36459/ using document root <none> and password file <none>
I20260925 19:13:39.313058 17150 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:39.313097 17150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:39.313190 17150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:39.314160 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "8d3d4bfada5b4d61a550b7c05ba2320c"
format_stamp: "Formatted at 2026-09-25 19:13:39 on dist-test-slave-fcdn"
server_key: "373fa376c1cdc8a946c0eba8d0239250"
server_key_iv: "24cd56eda1acff0aef7c67b4170861f3"
server_key_version: "encryptionkey@0"
I20260925 19:13:39.314440 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8d3d4bfada5b4d61a550b7c05ba2320c"
format_stamp: "Formatted at 2026-09-25 19:13:39 on dist-test-slave-fcdn"
server_key: "373fa376c1cdc8a946c0eba8d0239250"
server_key_iv: "24cd56eda1acff0aef7c67b4170861f3"
server_key_version: "encryptionkey@0"
I20260925 19:13:39.315706 17150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:13:39.316381 17165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:39.316548 17150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:39.316636 17150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "8d3d4bfada5b4d61a550b7c05ba2320c"
format_stamp: "Formatted at 2026-09-25 19:13:39 on dist-test-slave-fcdn"
server_key: "373fa376c1cdc8a946c0eba8d0239250"
server_key_iv: "24cd56eda1acff0aef7c67b4170861f3"
server_key_version: "encryptionkey@0"
I20260925 19:13:39.316694 17150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:39.333464 17150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:39.335750 17150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:39.335861 17150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:39.339470 17150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34715
I20260925 19:13:39.339535 17217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34715 every 8 connection(s)
I20260925 19:13:39.339860 17150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:39.340719 17218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:39.342895 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c: Bootstrap starting.
I20260925 19:13:39.343452 17218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:39.343801 17218 log.cc:824] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:39.344457 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c: No bootstrap required, opened a new log
I20260925 19:13:39.345573 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17150
I20260925 19:13:39.345700 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:39.345865 28345 external_mini_cluster.cc:1428] Setting key 1d15895cebe7e2836ceac182fa09b87a
I20260925 19:13:39.346283 17218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } }
I20260925 19:13:39.346464 17218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:39.346504 17218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3d4bfada5b4d61a550b7c05ba2320c, State: Initialized, Role: FOLLOWER
I20260925 19:13:39.346678 17218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } }
I20260925 19:13:39.346781 17218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:39.346824 17218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:39.346872 17218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:39.347791 17218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } }
I20260925 19:13:39.347908 17218 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3d4bfada5b4d61a550b7c05ba2320c; no voters:
I20260925 19:13:39.348083 17218 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:39.348116 17223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:39.348234 17223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [term 1 LEADER]: Becoming Leader. State: Replica: 8d3d4bfada5b4d61a550b7c05ba2320c, State: Running, Role: LEADER
I20260925 19:13:39.348340 17218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:39.348363 17223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } }
I20260925 19:13:39.348721 17223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3d4bfada5b4d61a550b7c05ba2320c. Latest consensus state: current_term: 1 leader_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } } }
I20260925 19:13:39.348953 17223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:39.348876 17224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d4bfada5b4d61a550b7c05ba2320c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34715 } } }
I20260925 19:13:39.349414 17224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:39.349586 17232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:39.350119 17232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:39.351519 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:39.346396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:42905 (local address 127.0.0.1:34715)
0925 19:13:39.346620 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:39.346629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:39.346662 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:13:39.346750 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:39.346756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:39.346824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:39.346957 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:39.347186 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:39.348269 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:39.349366 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:39.349996 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:39.350113 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:39.350834 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:39.350837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:39.350838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:39.351008 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:39.351038 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:39.351048 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:39.351155 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:39.351241 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:13:39.351270 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260925 19:13:39.351904 17232 catalog_manager.cc:1347] Generated new cluster ID: 069e298cefa7480d91e30a3f20eee740
I20260925 19:13:39.351960 17232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:39.352438 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:39.356534 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:39.353732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:54149 (local address 127.0.0.1:34715)
0925 19:13:39.353770 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:39.353771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:39.353873 (+ 102us) server_negotiation.cc:407] Connection header received
0925 19:13:39.353941 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:39.353942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:39.353952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:39.354007 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:39.354139 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:39.354593 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:39.355305 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:39.355547 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:39.355644 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:39.356296 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:39.356297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:39.356298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:39.356302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:39.356315 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:39.356319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:39.356371 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:39.356431 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 19:13:39.356443 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:13:39.358136 17232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:39.358825 17232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:39.362576 17232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3d4bfada5b4d61a550b7c05ba2320c: Generated new TSK 0
I20260925 19:13:39.362761 17232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:39.368873 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17150
W20260925 19:13:39.373062 17135 connection.cc:535] client connection to 127.0.0.1:34715 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34715): Connection reset by peer (error 104)
2026-09-25T19:13:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:39 dist-test-slave-fcdn krb5kdc[17249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:39 dist-test-slave-fcdn krb5kdc[17249](info): set up 2 sockets
Sep 25 19:13:39 dist-test-slave-fcdn krb5kdc[17249](info): commencing operation
krb5kdc: starting...
W20260925 19:13:40.411850 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:40 dist-test-slave-fcdn krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:13:40.527124 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:40.606395 17262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:40.606583 17262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:40.606606 17262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:40.607815 17262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:40.607853 17262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:40.607867 17262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:40.607880 17262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:40.607937 17262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:40.609467 17262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:40.609711 17262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:40.609923 17262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:40.614708 17268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:40.614786 17262 server_base.cc:1034] running on GCE node
W20260925 19:13:40.614760 17267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:40.614708 17270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:40.615176 17262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:40.615489 17262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:40.616681 17262 hybrid_clock.cc:648] HybridClock initialized: now 1790363620616661 us; error 31 us; skew 500 ppm
Sep 25 19:13:40 dist-test-slave-fcdn krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:40.619035 17262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:13:40.619452 17262 webserver.cc:460] Webserver started at http://127.0.0.1:40775/ using document root <none> and password file <none>
I20260925 19:13:40.619740 17262 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:40.619809 17262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:40.619941 17262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:40.621040 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "d8e5602377be4449b0e302b057c3e0a8"
format_stamp: "Formatted at 2026-09-25 19:13:40 on dist-test-slave-fcdn"
server_key: "852b133715a3dd8866fd421ee43b6851"
server_key_iv: "0e464fe9a4066f532626e23d6cc3ade3"
server_key_version: "encryptionkey@0"
I20260925 19:13:40.621385 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "d8e5602377be4449b0e302b057c3e0a8"
format_stamp: "Formatted at 2026-09-25 19:13:40 on dist-test-slave-fcdn"
server_key: "852b133715a3dd8866fd421ee43b6851"
server_key_iv: "0e464fe9a4066f532626e23d6cc3ade3"
server_key_version: "encryptionkey@0"
I20260925 19:13:40.622664 17262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:40.623375 17277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:40.623559 17262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:40.623674 17262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "d8e5602377be4449b0e302b057c3e0a8"
format_stamp: "Formatted at 2026-09-25 19:13:40 on dist-test-slave-fcdn"
server_key: "852b133715a3dd8866fd421ee43b6851"
server_key_iv: "0e464fe9a4066f532626e23d6cc3ade3"
server_key_version: "encryptionkey@0"
I20260925 19:13:40.623755 17262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:40.626857 17262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:40.628685 17262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:40.628851 17262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:40.632566 17262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40855
I20260925 19:13:40.632619 17329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40855 every 8 connection(s)
I20260925 19:13:40.632932 17262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:40.633858 17330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:40.635980 17330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8: Bootstrap starting.
I20260925 19:13:40.636469 17330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:40.636698 17330 log.cc:824] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:40.637324 17330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8: No bootstrap required, opened a new log
I20260925 19:13:40.638782 17330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260925 19:13:40.638908 17330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:40.638929 17330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e5602377be4449b0e302b057c3e0a8, State: Initialized, Role: FOLLOWER
I20260925 19:13:40.639022 17330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260925 19:13:40.639068 17330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:40.639087 17330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:40.639127 17330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:40.639585 17330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260925 19:13:40.639762 17330 leader_election.cc:302] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e5602377be4449b0e302b057c3e0a8; no voters:
I20260925 19:13:40.639878 17330 leader_election.cc:288] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:40.639917 17333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:40.640090 17333 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [term 1 LEADER]: Becoming Leader. State: Replica: d8e5602377be4449b0e302b057c3e0a8, State: Running, Role: LEADER
I20260925 19:13:40.640105 17330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:40.640197 17333 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260925 19:13:40.640618 17335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e5602377be4449b0e302b057c3e0a8. Latest consensus state: current_term: 1 leader_uuid: "d8e5602377be4449b0e302b057c3e0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260925 19:13:40.640715 17335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:40.640769 17334 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e5602377be4449b0e302b057c3e0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e5602377be4449b0e302b057c3e0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260925 19:13:40.640852 17334 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:40.641144 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17262
I20260925 19:13:40.641186 17341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:40.641263 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:40.641458 28345 external_mini_cluster.cc:1428] Setting key af01391d3f89f7a24cd76834ce11427b
I20260925 19:13:40.641733 17341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:40.643184 17341 catalog_manager.cc:1347] Generated new cluster ID: ff8a695d79ca47d18ad51786649f1da3
I20260925 19:13:40.643239 17341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:40.644181 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:40.641971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:47729 (local address 127.0.0.1:40855)
0925 19:13:40.642605 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:40.642625 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:40.642656 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:13:40.642760 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:40.642766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:40.642811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:40.642942 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:40.643756 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:40.643759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:40.643759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:40.643762 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:40.643794 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:40.643804 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:40.643879 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:40.643953 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:13:40.643998 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":91,"threads_started":1}
I20260925 19:13:40.645054 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:40.648641 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:40.646372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:46291 (local address 127.0.0.1:40855)
0925 19:13:40.646408 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:40.646410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:40.647772 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 19:13:40.647847 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:40.647848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:40.647854 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:40.647908 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:40.648421 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:40.648422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:40.648422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:40.648427 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:40.648438 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:40.648442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:40.648491 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:40.648549 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:13:40.648564 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:13:40.650794 17341 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:40.650933 17341 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:40.657318 17341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e5602377be4449b0e302b057c3e0a8: Generated new TSK 0
I20260925 19:13:40.657497 17341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:40.662439 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17262
2026-09-25T19:13:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:40 dist-test-slave-fcdn krb5kdc[17361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:40 dist-test-slave-fcdn krb5kdc[17361](info): set up 2 sockets
Sep 25 19:13:40 dist-test-slave-fcdn krb5kdc[17361](info): commencing operation
krb5kdc: starting...
W20260925 19:13:41.687583 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:41 dist-test-slave-fcdn krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:41.812024 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:39723
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:41.892463 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:41.892666 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:41.892689 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:41.893844 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:41.893887 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:41.893901 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:41.893914 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:41.893926 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:41.895399 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:39723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:41.895673 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:41.895908 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:41.900749 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:41.900758 17374 server_base.cc:1034] running on GCE node
W20260925 19:13:41.900767 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:41.900758 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:41.901306 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:41.901611 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:41.902756 17374 hybrid_clock.cc:648] HybridClock initialized: now 1790363621902748 us; error 33 us; skew 500 ppm
Sep 25 19:13:41 dist-test-slave-fcdn krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363621, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:41.917605 17374 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:41.918059 17374 webserver.cc:460] Webserver started at http://127.27.174.126:42645/ using document root <none> and password file <none>
I20260925 19:13:41.918305 17374 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:41.918375 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:41.918525 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:41.919585 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "c44385847ad24bb08fef2dbc2cffe03e"
format_stamp: "Formatted at 2026-09-25 19:13:41 on dist-test-slave-fcdn"
server_key: "9a8bdd4484d20109085d889ec6190ec6"
server_key_iv: "8960c380626e4ecdd552c322dc9e4f75"
server_key_version: "encryptionkey@0"
I20260925 19:13:41.920035 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "c44385847ad24bb08fef2dbc2cffe03e"
format_stamp: "Formatted at 2026-09-25 19:13:41 on dist-test-slave-fcdn"
server_key: "9a8bdd4484d20109085d889ec6190ec6"
server_key_iv: "8960c380626e4ecdd552c322dc9e4f75"
server_key_version: "encryptionkey@0"
I20260925 19:13:41.921407 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:41.922217 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:41.922423 17374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:41.922524 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "c44385847ad24bb08fef2dbc2cffe03e"
format_stamp: "Formatted at 2026-09-25 19:13:41 on dist-test-slave-fcdn"
server_key: "9a8bdd4484d20109085d889ec6190ec6"
server_key_iv: "8960c380626e4ecdd552c322dc9e4f75"
server_key_version: "encryptionkey@0"
I20260925 19:13:41.922612 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:41.940726 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:41.943113 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:41.943280 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:41.947100 17374 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:39723
I20260925 19:13:41.947127 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:39723 every 8 connection(s)
I20260925 19:13:41.947485 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:41.948369 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:41.950639 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e: Bootstrap starting.
I20260925 19:13:41.951208 17442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:41.951535 17442 log.cc:824] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:41.952224 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e: No bootstrap required, opened a new log
I20260925 19:13:41.953799 17442 raft_consensus.cc:348] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } }
I20260925 19:13:41.953931 17442 raft_consensus.cc:374] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:41.953954 17442 raft_consensus.cc:729] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44385847ad24bb08fef2dbc2cffe03e, State: Initialized, Role: FOLLOWER
I20260925 19:13:41.954126 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } }
I20260925 19:13:41.954188 17442 raft_consensus.cc:388] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:41.954209 17442 raft_consensus.cc:482] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:41.954280 17442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:41.955111 17442 raft_consensus.cc:504] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } }
I20260925 19:13:41.955194 17442 leader_election.cc:302] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c44385847ad24bb08fef2dbc2cffe03e; no voters:
I20260925 19:13:41.955402 17442 leader_election.cc:288] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:41.955435 17445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:41.955636 17445 raft_consensus.cc:686] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [term 1 LEADER]: Becoming Leader. State: Replica: c44385847ad24bb08fef2dbc2cffe03e, State: Running, Role: LEADER
I20260925 19:13:41.955713 17442 sys_catalog.cc:564] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:41.955778 17445 consensus_queue.cc:237] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } }
I20260925 19:13:41.956144 17446 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c44385847ad24bb08fef2dbc2cffe03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } } }
I20260925 19:13:41.956157 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44385847ad24bb08fef2dbc2cffe03e. Latest consensus state: current_term: 1 leader_uuid: "c44385847ad24bb08fef2dbc2cffe03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44385847ad24bb08fef2dbc2cffe03e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39723 } } }
I20260925 19:13:41.956269 17446 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:41.956287 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:41.956482 17454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:41.957049 17454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:41.957219 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17374
I20260925 19:13:41.957338 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:41.957516 28345 external_mini_cluster.cc:1428] Setting key b0a1f76eaef82b232277a2b4ec3324ec
I20260925 19:13:41.958637 17454 catalog_manager.cc:1347] Generated new cluster ID: 532e80db879a4172bd911d16a7041abc
I20260925 19:13:41.958688 17454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:13:41 dist-test-slave-fcdn krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:41.969332 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:41.958026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39895 (local address 127.27.174.126:39723)
0925 19:13:41.958210 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:41.958222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:41.958410 (+ 188us) server_negotiation.cc:407] Connection header received
0925 19:13:41.958583 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:41.958591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:41.958650 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:41.958789 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:41.959559 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:41.960948 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:41.962029 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:41.962604 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:41.962697 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:41.965620 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:41.965625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:41.965625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:41.965638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:41.967636 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:41.968062 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:41.968066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:41.968066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:41.968173 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:41.968397 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:41.968398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:41.968399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:41.968719 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:41.968876 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:41.968997 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:13:41.969098 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260925 19:13:41.970456 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:41.978495 17454 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:41.979249 17454 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:41.979461 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:41.971964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45111 (local address 127.27.174.126:39723)
0925 19:13:41.972001 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:41.972003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:41.972203 (+ 200us) server_negotiation.cc:407] Connection header received
0925 19:13:41.972234 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:41.972235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:41.972245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:41.972303 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:41.972854 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:41.973627 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:41.974600 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:41.974932 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:41.975002 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:41.976352 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:41.976353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:41.976354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:41.976360 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:41.978006 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:41.978583 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:41.978584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:41.978585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:41.978642 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:41.978877 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:41.978878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:41.978878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:41.979028 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:41.979143 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:41.979290 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:13:41.979356 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:41.986783 17454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c44385847ad24bb08fef2dbc2cffe03e: Generated new TSK 0
I20260925 19:13:41.986999 17454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:41.990003 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17374
W20260925 19:13:41.993846 17359 connection.cc:535] client connection to 127.27.174.126:39723 recv error: Network error: failed to read from TLS socket (remote: 127.27.174.126:39723): Connection reset by peer (error 104)
2026-09-25T19:13:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:42 dist-test-slave-fcdn krb5kdc[17473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:42 dist-test-slave-fcdn krb5kdc[17473](info): set up 2 sockets
Sep 25 19:13:42 dist-test-slave-fcdn krb5kdc[17473](info): commencing operation
krb5kdc: starting...
W20260925 19:13:43.027693 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:43 dist-test-slave-fcdn krb5kdc[17473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:43.147068 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45299
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:43.227246 17486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:43.227505 17486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:43.227543 17486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:43.228693 17486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:43.228767 17486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:43.228790 17486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:43.228811 17486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:43.228840 17486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:43.230301 17486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:43.230583 17486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:43.230815 17486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:43.235469 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:43.235514 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:43.235703 17486 server_base.cc:1034] running on GCE node
W20260925 19:13:43.235481 17494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:43.235947 17486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:43.236192 17486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:43.237360 17486 hybrid_clock.cc:648] HybridClock initialized: now 1790363623237339 us; error 53 us; skew 500 ppm
Sep 25 19:13:43 dist-test-slave-fcdn krb5kdc[17473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:43.239866 17486 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:43.240298 17486 webserver.cc:460] Webserver started at http://127.27.174.126:34919/ using document root <none> and password file <none>
I20260925 19:13:43.240545 17486 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:43.240621 17486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:43.240767 17486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:43.241818 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30"
format_stamp: "Formatted at 2026-09-25 19:13:43 on dist-test-slave-fcdn"
server_key: "d1381adca531df71ad7c1916c602dec3"
server_key_iv: "c424792453c53f693fbf5a1eda1e1785"
server_key_version: "encryptionkey@0"
I20260925 19:13:43.242165 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30"
format_stamp: "Formatted at 2026-09-25 19:13:43 on dist-test-slave-fcdn"
server_key: "d1381adca531df71ad7c1916c602dec3"
server_key_iv: "c424792453c53f693fbf5a1eda1e1785"
server_key_version: "encryptionkey@0"
I20260925 19:13:43.243475 17486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:13:43.244210 17501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:43.244377 17486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:43.244504 17486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30"
format_stamp: "Formatted at 2026-09-25 19:13:43 on dist-test-slave-fcdn"
server_key: "d1381adca531df71ad7c1916c602dec3"
server_key_iv: "c424792453c53f693fbf5a1eda1e1785"
server_key_version: "encryptionkey@0"
I20260925 19:13:43.244586 17486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:43.260926 17486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:43.262665 17486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:43.262825 17486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:43.266716 17486 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45299
I20260925 19:13:43.266767 17553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45299 every 8 connection(s)
I20260925 19:13:43.267127 17486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:43.268045 17554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:43.270200 17554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30: Bootstrap starting.
I20260925 19:13:43.270716 17554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:43.270941 17554 log.cc:824] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:43.271579 17554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30: No bootstrap required, opened a new log
I20260925 19:13:43.271832 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17486
I20260925 19:13:43.271943 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:43.272142 28345 external_mini_cluster.cc:1428] Setting key fb1230f68f1bf55b8756333cec28f4e9
I20260925 19:13:43.273690 17554 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } }
I20260925 19:13:43.273855 17554 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:43.273895 17554 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f175f6dd824f0aa8abd5acf2c4ec30, State: Initialized, Role: FOLLOWER
I20260925 19:13:43.274088 17554 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } }
I20260925 19:13:43.274200 17554 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:43.274245 17554 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:43.274296 17554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:43.274811 17554 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } }
I20260925 19:13:43.274925 17554 leader_election.cc:302] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f175f6dd824f0aa8abd5acf2c4ec30; no voters:
I20260925 19:13:43.275117 17554 leader_election.cc:288] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:43.275137 17559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:43.275305 17559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [term 1 LEADER]: Becoming Leader. State: Replica: d2f175f6dd824f0aa8abd5acf2c4ec30, State: Running, Role: LEADER
I20260925 19:13:43.275416 17554 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:43.275437 17559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } }
I20260925 19:13:43.275813 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f175f6dd824f0aa8abd5acf2c4ec30. Latest consensus state: current_term: 1 leader_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } } }
I20260925 19:13:43.275918 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:43.276371 17565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:43.276887 17565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:43.277788 17560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f175f6dd824f0aa8abd5acf2c4ec30" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45299 } } }
I20260925 19:13:43.277938 17560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:43.278473 17565 catalog_manager.cc:1347] Generated new cluster ID: 231fdc83b73145d8a98f9d18154d2b59
I20260925 19:13:43.278527 17565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:43.278777 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:43.272628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32843 (local address 127.27.174.126:45299)
0925 19:13:43.272787 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:43.272796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:43.272829 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:13:43.273673 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:43.273679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:43.273723 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:43.273832 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:43.274122 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:43.275537 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:43.276679 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:43.277282 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:43.277371 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:43.278126 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:43.278130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:43.278131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:43.278319 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:43.278349 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:43.278359 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:43.278466 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:43.278547 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:13:43.278573 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 19:13:43.279727 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:43.284027 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:43.281101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49183 (local address 127.27.174.126:45299)
0925 19:13:43.281137 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:43.281139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:43.281229 (+ 90us) server_negotiation.cc:407] Connection header received
0925 19:13:43.281272 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:43.281273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:43.281281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:43.281343 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:43.281436 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:43.282128 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:43.282810 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:43.283097 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:43.283156 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:43.283805 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:43.283806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:43.283806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:43.283810 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:43.283822 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:43.283826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:43.283880 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:43.283948 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 19:13:43.283958 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":20}
I20260925 19:13:43.285895 17565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:43.286499 17565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:43.291884 17565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f175f6dd824f0aa8abd5acf2c4ec30: Generated new TSK 0
I20260925 19:13:43.292052 17565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:43.300957 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17486
W20260925 19:13:43.305248 17471 connection.cc:535] client connection to 127.27.174.126:45299 recv error: Network error: failed to read from TLS socket (remote: 127.27.174.126:45299): Connection reset by peer (error 104)
2026-09-25T19:13:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:43 dist-test-slave-fcdn krb5kdc[17585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:43 dist-test-slave-fcdn krb5kdc[17585](info): set up 2 sockets
Sep 25 19:13:43 dist-test-slave-fcdn krb5kdc[17585](info): commencing operation
krb5kdc: starting...
W20260925 19:13:44.343725 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:44 dist-test-slave-fcdn krb5kdc[17585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:44.462947 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33933
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:44.545326 17598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:44.545545 17598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:44.545569 17598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:44.546782 17598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:44.546829 17598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:44.546844 17598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:44.546856 17598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:44.546868 17598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:44.548470 17598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:44.548734 17598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:44.548978 17598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:44.553885 17603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:44.553885 17604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:44.553959 17606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:44.554340 17598 server_base.cc:1034] running on GCE node
I20260925 19:13:44.554563 17598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:44.554811 17598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:44.555969 17598 hybrid_clock.cc:648] HybridClock initialized: now 1790363624555955 us; error 28 us; skew 500 ppm
Sep 25 19:13:44 dist-test-slave-fcdn krb5kdc[17585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:44.558741 17598 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:44.559178 17598 webserver.cc:460] Webserver started at http://127.27.174.126:43049/ using document root <none> and password file <none>
I20260925 19:13:44.559424 17598 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:44.559473 17598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:44.559643 17598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:44.560781 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "77a92f4738e6451f9847ad343f1119d5"
format_stamp: "Formatted at 2026-09-25 19:13:44 on dist-test-slave-fcdn"
server_key: "475bb2635a04f7164e9ebbae6e97ce33"
server_key_iv: "9a9b59a80645a2f30b4d19a01ec4a73d"
server_key_version: "encryptionkey@0"
I20260925 19:13:44.561138 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "77a92f4738e6451f9847ad343f1119d5"
format_stamp: "Formatted at 2026-09-25 19:13:44 on dist-test-slave-fcdn"
server_key: "475bb2635a04f7164e9ebbae6e97ce33"
server_key_iv: "9a9b59a80645a2f30b4d19a01ec4a73d"
server_key_version: "encryptionkey@0"
I20260925 19:13:44.562453 17598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:44.563160 17613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:44.563364 17598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:44.563476 17598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "77a92f4738e6451f9847ad343f1119d5"
format_stamp: "Formatted at 2026-09-25 19:13:44 on dist-test-slave-fcdn"
server_key: "475bb2635a04f7164e9ebbae6e97ce33"
server_key_iv: "9a9b59a80645a2f30b4d19a01ec4a73d"
server_key_version: "encryptionkey@0"
I20260925 19:13:44.563555 17598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:44.566821 17598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:44.569175 17598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:44.569348 17598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:44.573117 17598 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33933
I20260925 19:13:44.573163 17665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33933 every 8 connection(s)
I20260925 19:13:44.573484 17598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:44.574388 17666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:44.576646 17666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5: Bootstrap starting.
I20260925 19:13:44.577193 17666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:44.577497 17666 log.cc:824] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:44.577605 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17598
I20260925 19:13:44.577736 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:44.577904 28345 external_mini_cluster.cc:1428] Setting key 6d719849702edd3c64b4918444bde419
I20260925 19:13:44.578347 17666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5: No bootstrap required, opened a new log
I20260925 19:13:44.580531 17666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } }
I20260925 19:13:44.580765 17666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:44.580835 17666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a92f4738e6451f9847ad343f1119d5, State: Initialized, Role: FOLLOWER
I20260925 19:13:44.580960 17666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } }
I20260925 19:13:44.581063 17666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:44.581100 17666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:44.581115 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:44.578387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33959 (local address 127.27.174.126:33933)
0925 19:13:44.578580 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:44.578588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:44.578622 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:13:44.579639 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:44.579648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:44.579690 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:44.579828 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:44.580501 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:44.580504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:44.580504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:44.580509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:44.580552 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:44.580566 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:44.580646 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:44.580684 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:13:44.580723 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260925 19:13:44.581153 17666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:44.581853 17666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } }
I20260925 19:13:44.581952 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:44.581964 17666 leader_election.cc:302] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77a92f4738e6451f9847ad343f1119d5; no voters:
I20260925 19:13:44.582119 17666 leader_election.cc:288] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:44.582160 17671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:44.582315 17671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [term 1 LEADER]: Becoming Leader. State: Replica: 77a92f4738e6451f9847ad343f1119d5, State: Running, Role: LEADER
I20260925 19:13:44.582370 17666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:44.582445 17671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } }
I20260925 19:13:44.582783 17673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77a92f4738e6451f9847ad343f1119d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } } }
I20260925 19:13:44.582948 17673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:44.582767 17671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77a92f4738e6451f9847ad343f1119d5. Latest consensus state: current_term: 1 leader_uuid: "77a92f4738e6451f9847ad343f1119d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a92f4738e6451f9847ad343f1119d5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33933 } } }
I20260925 19:13:44.583169 17671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:44.583334 17681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:44.583894 17681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:44.585384 17681 catalog_manager.cc:1347] Generated new cluster ID: 088885baf09d47f485a8d4ba7e110930
I20260925 19:13:44.585435 17681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:44.593050 17681 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:44.593200 17681 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:44.593962 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:44.584856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39135 (local address 127.27.174.126:33933)
0925 19:13:44.584886 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:44.584888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:44.592877 (+ 7989us) server_negotiation.cc:407] Connection header received
0925 19:13:44.592994 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:44.592996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:44.593001 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:44.593065 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:44.593737 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:44.593739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:44.593739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:44.593743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:44.593756 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:44.593760 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:44.593824 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:44.593882 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:13:44.593897 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:13:44.607285 17681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77a92f4738e6451f9847ad343f1119d5: Generated new TSK 0
I20260925 19:13:44.607479 17681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:44.609864 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17598
2026-09-25T19:13:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:44 dist-test-slave-fcdn krb5kdc[17697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:44 dist-test-slave-fcdn krb5kdc[17697](info): set up 2 sockets
Sep 25 19:13:44 dist-test-slave-fcdn krb5kdc[17697](info): commencing operation
krb5kdc: starting...
W20260925 19:13:45.647641 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:45.763275 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37633
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:45.844449 17710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:45.844702 17710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:45.844738 17710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:45.845923 17710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:45.845991 17710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:45.846014 17710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:45.846036 17710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:45.846066 17710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:45.847499 17710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:45.847786 17710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:45.848003 17710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:45.852667 17715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:45.852702 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:45.852731 17716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:45.852948 17710 server_base.cc:1034] running on GCE node
I20260925 19:13:45.853170 17710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:45.853394 17710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:45.854537 17710 hybrid_clock.cc:648] HybridClock initialized: now 1790363625854519 us; error 33 us; skew 500 ppm
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363625, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:45.857007 17710 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:45.857470 17710 webserver.cc:460] Webserver started at http://127.27.174.126:44275/ using document root <none> and password file <none>
I20260925 19:13:45.857692 17710 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:45.857735 17710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:45.857872 17710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:45.858863 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb"
format_stamp: "Formatted at 2026-09-25 19:13:45 on dist-test-slave-fcdn"
server_key: "42dd68884d33dcfa9b72a4fed6205315"
server_key_iv: "d883523580ec621fa6e94ad3344b05bf"
server_key_version: "encryptionkey@0"
I20260925 19:13:45.859217 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb"
format_stamp: "Formatted at 2026-09-25 19:13:45 on dist-test-slave-fcdn"
server_key: "42dd68884d33dcfa9b72a4fed6205315"
server_key_iv: "d883523580ec621fa6e94ad3344b05bf"
server_key_version: "encryptionkey@0"
I20260925 19:13:45.860591 17710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:45.861270 17725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:45.861455 17710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:45.861545 17710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb"
format_stamp: "Formatted at 2026-09-25 19:13:45 on dist-test-slave-fcdn"
server_key: "42dd68884d33dcfa9b72a4fed6205315"
server_key_iv: "d883523580ec621fa6e94ad3344b05bf"
server_key_version: "encryptionkey@0"
I20260925 19:13:45.861618 17710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:45.878988 17710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:45.881385 17710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:45.881562 17710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:45.885661 17710 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:37633
I20260925 19:13:45.885694 17777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:37633 every 8 connection(s)
I20260925 19:13:45.886047 17710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:45.886914 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:45.887521 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17710
I20260925 19:13:45.887657 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:45.887849 28345 external_mini_cluster.cc:1428] Setting key 68f742a26719f6d0b1588ed4fc0a793f
I20260925 19:13:45.890177 17778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb: Bootstrap starting.
I20260925 19:13:45.890755 17778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:45.891034 17778 log.cc:824] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:45.891808 17778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb: No bootstrap required, opened a new log
I20260925 19:13:45.893643 17778 raft_consensus.cc:348] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } }
I20260925 19:13:45.893800 17778 raft_consensus.cc:374] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:45.893848 17778 raft_consensus.cc:729] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcd7ae48f2c43ccbd20cf7a07e170eb, State: Initialized, Role: FOLLOWER
I20260925 19:13:45.893980 17778 consensus_queue.cc:260] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } }
I20260925 19:13:45.894068 17778 raft_consensus.cc:388] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:45.894107 17778 raft_consensus.cc:482] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:45.894155 17778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:45.894732 17778 raft_consensus.cc:504] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } }
I20260925 19:13:45.894848 17778 leader_election.cc:302] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abcd7ae48f2c43ccbd20cf7a07e170eb; no voters:
I20260925 19:13:45.895048 17778 leader_election.cc:288] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:45.895076 17783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:45.895181 17783 raft_consensus.cc:686] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [term 1 LEADER]: Becoming Leader. State: Replica: abcd7ae48f2c43ccbd20cf7a07e170eb, State: Running, Role: LEADER
I20260925 19:13:45.895311 17783 consensus_queue.cc:237] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } }
I20260925 19:13:45.895315 17778 sys_catalog.cc:564] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:45.895757 17785 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader abcd7ae48f2c43ccbd20cf7a07e170eb. Latest consensus state: current_term: 1 leader_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } } }
I20260925 19:13:45.895839 17785 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:45.896042 17784 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd7ae48f2c43ccbd20cf7a07e170eb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37633 } } }
I20260925 19:13:45.896364 17784 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:45.896560 17787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:45.897022 17787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:45.898509 17787 catalog_manager.cc:1347] Generated new cluster ID: 686bcbcddbad40ae92e6ea65ecdee4a7
I20260925 19:13:45.898561 17787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:45.898706 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:45.888334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56301 (local address 127.27.174.126:37633)
0925 19:13:45.888520 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:45.888528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:45.888553 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:13:45.888938 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:45.888945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:45.888989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:45.889102 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:45.889895 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:45.890816 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:45.891887 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:45.892292 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:45.892428 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:45.894818 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:45.894821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:45.894822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:45.894834 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:45.897024 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:45.897425 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:45.897428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:45.897429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:45.897538 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:45.897734 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:45.897735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:45.897735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:45.898028 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:45.898172 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:45.898275 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:13:45.898512 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260925 19:13:45.899382 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:45.907895 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:45.900840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43597 (local address 127.27.174.126:37633)
0925 19:13:45.900870 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:45.900872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:45.900971 (+ 99us) server_negotiation.cc:407] Connection header received
0925 19:13:45.901028 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:45.901029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:45.901037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:45.901090 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:45.901715 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:45.902401 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:45.903497 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:45.903826 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:45.903884 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:45.905204 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:45.905205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:45.905206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:45.905212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:45.906717 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:45.907093 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:45.907094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:45.907094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:45.907148 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:45.907358 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:45.907358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:45.907358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:45.907477 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:45.907556 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:45.907746 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:13:45.907813 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 19:13:45.915496 17787 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:45.916214 17787 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:45.920125 17787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abcd7ae48f2c43ccbd20cf7a07e170eb: Generated new TSK 0
I20260925 19:13:45.920296 17787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:45.922739 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17710
2026-09-25T19:13:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17809](info): set up 2 sockets
Sep 25 19:13:45 dist-test-slave-fcdn krb5kdc[17809](info): commencing operation
krb5kdc: starting...
W20260925 19:13:46.952263 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:47 dist-test-slave-fcdn krb5kdc[17809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:47.070194 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44877
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:47.149293 17822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:47.149535 17822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:47.149572 17822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:47.150741 17822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:47.150810 17822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:47.150836 17822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:47.150857 17822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:47.150885 17822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:47.152420 17822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:47.152691 17822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:47.152916 17822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:47.157721 17828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:47.157783 17830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:47.157721 17827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:47.157966 17822 server_base.cc:1034] running on GCE node
I20260925 19:13:47.158200 17822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:47.158447 17822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:47.159627 17822 hybrid_clock.cc:648] HybridClock initialized: now 1790363627159579 us; error 39 us; skew 500 ppm
Sep 25 19:13:47 dist-test-slave-fcdn krb5kdc[17809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363627, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:47.169154 17822 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:47.169624 17822 webserver.cc:460] Webserver started at http://127.27.174.126:39133/ using document root <none> and password file <none>
I20260925 19:13:47.169865 17822 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:47.169911 17822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:47.170068 17822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:47.171125 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "0bd45aa4401f45fcb30672e073382fdd"
format_stamp: "Formatted at 2026-09-25 19:13:47 on dist-test-slave-fcdn"
server_key: "eed6e7ba15f92b9869a54975011091fb"
server_key_iv: "49a39618080f7ffbfe2c20d0d67f4a98"
server_key_version: "encryptionkey@0"
I20260925 19:13:47.171487 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "0bd45aa4401f45fcb30672e073382fdd"
format_stamp: "Formatted at 2026-09-25 19:13:47 on dist-test-slave-fcdn"
server_key: "eed6e7ba15f92b9869a54975011091fb"
server_key_iv: "49a39618080f7ffbfe2c20d0d67f4a98"
server_key_version: "encryptionkey@0"
I20260925 19:13:47.172803 17822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:47.173583 17837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:47.173794 17822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:47.173893 17822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "0bd45aa4401f45fcb30672e073382fdd"
format_stamp: "Formatted at 2026-09-25 19:13:47 on dist-test-slave-fcdn"
server_key: "eed6e7ba15f92b9869a54975011091fb"
server_key_iv: "49a39618080f7ffbfe2c20d0d67f4a98"
server_key_version: "encryptionkey@0"
I20260925 19:13:47.173962 17822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:47.184422 17822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:47.186622 17822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:47.186728 17822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:47.190505 17822 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44877
I20260925 19:13:47.190582 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44877 every 8 connection(s)
I20260925 19:13:47.190915 17822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:47.191824 17890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:47.194018 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd: Bootstrap starting.
I20260925 19:13:47.194666 17890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:47.194671 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17822
I20260925 19:13:47.194787 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:47.194981 28345 external_mini_cluster.cc:1428] Setting key c4fccd903fd301b2438f635f2b3abbd1
I20260925 19:13:47.195012 17890 log.cc:824] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:47.195904 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd: No bootstrap required, opened a new log
I20260925 19:13:47.197719 17890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } }
I20260925 19:13:47.197873 17890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:47.197932 17890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd45aa4401f45fcb30672e073382fdd, State: Initialized, Role: FOLLOWER
I20260925 19:13:47.198056 17890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } }
I20260925 19:13:47.198143 17890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:47.198184 17890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:47.198239 17890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:47.198793 17890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } }
I20260925 19:13:47.198920 17890 leader_election.cc:302] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bd45aa4401f45fcb30672e073382fdd; no voters:
I20260925 19:13:47.199106 17890 leader_election.cc:288] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:47.199148 17895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:47.199297 17895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [term 1 LEADER]: Becoming Leader. State: Replica: 0bd45aa4401f45fcb30672e073382fdd, State: Running, Role: LEADER
I20260925 19:13:47.199391 17895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } }
I20260925 19:13:47.199425 17890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:47.199723 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd45aa4401f45fcb30672e073382fdd. Latest consensus state: current_term: 1 leader_uuid: "0bd45aa4401f45fcb30672e073382fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } } }
I20260925 19:13:47.199733 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bd45aa4401f45fcb30672e073382fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd45aa4401f45fcb30672e073382fdd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44877 } } }
I20260925 19:13:47.199807 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:47.199812 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:47.200014 17900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:47.200505 17900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:47.200759 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:47.195466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33479 (local address 127.27.174.126:44877)
0925 19:13:47.195667 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:47.195678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:47.195708 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:13:47.196180 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:47.196187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:47.196233 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:47.196386 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:47.196601 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:47.197466 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:47.198497 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:47.198925 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:47.199062 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:47.199799 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:47.199803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:47.199804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:47.200061 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:47.200105 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:47.200124 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:47.200171 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:47.200430 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 19:13:47.200508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260925 19:13:47.202075 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:47.203480 17900 catalog_manager.cc:1347] Generated new cluster ID: 09e0c7bc2a9043d899d0ec33d0b1b9d6
I20260925 19:13:47.203534 17900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:47.211186 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:47.203566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40353 (local address 127.27.174.126:44877)
0925 19:13:47.203627 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:47.203629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:47.207869 (+ 4240us) server_negotiation.cc:407] Connection header received
0925 19:13:47.207949 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:47.207950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:47.207959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:47.208015 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:47.208148 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:47.208847 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:47.209888 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:47.210187 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:47.210250 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:47.210948 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:47.210949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:47.210949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:47.210953 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:47.210968 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:47.210972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:47.211021 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:47.211092 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:13:47.211111 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":20}
I20260925 19:13:47.216535 17900 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:47.217036 17900 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:47.233886 17900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bd45aa4401f45fcb30672e073382fdd: Generated new TSK 0
I20260925 19:13:47.234098 17900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:47.237030 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17822
2026-09-25T19:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:47 dist-test-slave-fcdn krb5kdc[17921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:47 dist-test-slave-fcdn krb5kdc[17921](info): set up 2 sockets
Sep 25 19:13:47 dist-test-slave-fcdn krb5kdc[17921](info): commencing operation
krb5kdc: starting...
W20260925 19:13:48.271692 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:48 dist-test-slave-fcdn krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:48.389968 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41689
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:48.487072 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:48.487293 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:48.487362 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:48.488642 17934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:48.488688 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:48.488703 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:48.488755 17934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:48.488778 17934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:48.490284 17934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:48.490546 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:48.490785 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:48.495671 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:48.495793 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:48.495718 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:48.496031 17934 server_base.cc:1034] running on GCE node
I20260925 19:13:48.496259 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:48.496544 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:48.497711 17934 hybrid_clock.cc:648] HybridClock initialized: now 1790363628497699 us; error 26 us; skew 500 ppm
Sep 25 19:13:48 dist-test-slave-fcdn krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:48.500165 17934 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:48.500655 17934 webserver.cc:460] Webserver started at http://127.27.174.126:35391/ using document root <none> and password file <none>
I20260925 19:13:48.500893 17934 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:48.500962 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:48.501117 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:48.502213 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "ea1122eb08eb477c9550261f5c356e66"
format_stamp: "Formatted at 2026-09-25 19:13:48 on dist-test-slave-fcdn"
server_key: "87f73b7b731844b87882be6f193250cb"
server_key_iv: "9c01dae67d78c21e2b5ab2f703878d7a"
server_key_version: "encryptionkey@0"
I20260925 19:13:48.502579 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ea1122eb08eb477c9550261f5c356e66"
format_stamp: "Formatted at 2026-09-25 19:13:48 on dist-test-slave-fcdn"
server_key: "87f73b7b731844b87882be6f193250cb"
server_key_iv: "9c01dae67d78c21e2b5ab2f703878d7a"
server_key_version: "encryptionkey@0"
I20260925 19:13:48.504208 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:48.505100 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:48.505347 17934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:13:48.505460 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "ea1122eb08eb477c9550261f5c356e66"
format_stamp: "Formatted at 2026-09-25 19:13:48 on dist-test-slave-fcdn"
server_key: "87f73b7b731844b87882be6f193250cb"
server_key_iv: "9c01dae67d78c21e2b5ab2f703878d7a"
server_key_version: "encryptionkey@0"
I20260925 19:13:48.505546 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:48.508967 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:48.511164 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:48.511363 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:48.515343 17934 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41689
I20260925 19:13:48.515379 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41689 every 8 connection(s)
I20260925 19:13:48.515771 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:48.516733 18002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:48.518962 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66: Bootstrap starting.
I20260925 19:13:48.519547 18002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:48.519865 18002 log.cc:824] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:48.520591 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66: No bootstrap required, opened a new log
I20260925 19:13:48.522147 18002 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } }
I20260925 19:13:48.522303 18002 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:13:48.522326 18002 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1122eb08eb477c9550261f5c356e66, State: Initialized, Role: FOLLOWER
I20260925 19:13:48.522487 18002 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } }
I20260925 19:13:48.522548 18002 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:13:48.522598 18002 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:13:48.522665 18002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:48.523515 18002 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } }
I20260925 19:13:48.523660 18002 leader_election.cc:302] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1122eb08eb477c9550261f5c356e66; no voters:
I20260925 19:13:48.523823 18002 leader_election.cc:288] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:13:48.523900 18005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:48.524071 18005 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [term 1 LEADER]: Becoming Leader. State: Replica: ea1122eb08eb477c9550261f5c356e66, State: Running, Role: LEADER
I20260925 19:13:48.524098 18002 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:48.524241 18005 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } }
I20260925 19:13:48.524653 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea1122eb08eb477c9550261f5c356e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } } }
I20260925 19:13:48.524753 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:48.524989 18014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:48.525158 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 17934
I20260925 19:13:48.525267 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:48.525467 28345 external_mini_cluster.cc:1428] Setting key addd115159326e9252a8944533187ae1
I20260925 19:13:48.525655 18014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:48.525967 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1122eb08eb477c9550261f5c356e66. Latest consensus state: current_term: 1 leader_uuid: "ea1122eb08eb477c9550261f5c356e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1122eb08eb477c9550261f5c356e66" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41689 } } }
I20260925 19:13:48.526365 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:48.527338 18014 catalog_manager.cc:1347] Generated new cluster ID: a1e962d5c89e4b959e40465346805f38
I20260925 19:13:48.527436 18014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:48.528229 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:48.526074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42121 (local address 127.27.174.126:41689)
0925 19:13:48.526481 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:48.526490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:48.526544 (+ 54us) server_negotiation.cc:407] Connection header received
0925 19:13:48.526659 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:48.526667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:48.526698 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:48.526839 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:48.527795 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:48.527802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:48.527802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:48.527810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:48.527850 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:48.527861 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:48.527936 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:48.528008 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 19:13:48.528050 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":82,"threads_started":1}
I20260925 19:13:48.529210 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:48.532738 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:48.530534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44469 (local address 127.27.174.126:41689)
0925 19:13:48.530570 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:48.530572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:48.531773 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 19:13:48.531852 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:48.531853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:48.531859 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:48.531913 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:48.532521 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:48.532522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:48.532522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:13:48.532526 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:48.532539 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:13:48.532543 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:48.532590 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:48.532642 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 19:13:48.532657 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:13:48.541072 18014 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:48.541244 18014 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:48.545868 18014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea1122eb08eb477c9550261f5c356e66: Generated new TSK 0
I20260925 19:13:48.546038 18014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:48.548164 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 17934
2026-09-25T19:13:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1311 ms)
[----------] 18 tests from AuthTokenIssuingTest (23579 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:48 dist-test-slave-fcdn krb5kdc[18033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:48 dist-test-slave-fcdn krb5kdc[18033](info): set up 2 sockets
Sep 25 19:13:48 dist-test-slave-fcdn krb5kdc[18033](info): commencing operation
krb5kdc: starting...
W20260925 19:13:49.579835 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:49.699404 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42597
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:49.780944 18046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:49.781179 18046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:49.781208 18046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:49.781277 18046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:49.782436 18046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:49.782503 18046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:49.782528 18046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:49.782560 18046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:49.782589 18046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:49.784113 18046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:49.784386 18046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:49.784607 18046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:49.789458 18051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:49.789476 18052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:49.789445 18054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:49.789770 18046 server_base.cc:1034] running on GCE node
I20260925 19:13:49.790135 18046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:49.790469 18046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:49.791680 18046 hybrid_clock.cc:648] HybridClock initialized: now 1790363629791661 us; error 39 us; skew 500 ppm
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:49.794771 18046 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:49.795141 18046 webserver.cc:460] Webserver started at http://127.27.174.126:44885/ using document root <none> and password file <none>
I20260925 19:13:49.795320 18046 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:49.795359 18046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:49.795444 18046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:49.796501 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "2f686039001940e9a6826ee538f4e31f"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "8e78c61e591c03e3fd2f8b5ebf71bab2"
server_key_iv: "57b9328092198f994cb60e70880fa8b4"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.796784 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "2f686039001940e9a6826ee538f4e31f"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "8e78c61e591c03e3fd2f8b5ebf71bab2"
server_key_iv: "57b9328092198f994cb60e70880fa8b4"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.798094 18046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:49.798766 18061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:49.798899 18046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:49.798965 18046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "2f686039001940e9a6826ee538f4e31f"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "8e78c61e591c03e3fd2f8b5ebf71bab2"
server_key_iv: "57b9328092198f994cb60e70880fa8b4"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.799015 18046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:49.810707 18046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:49.812976 18046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:49.813083 18046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:49.817003 18046 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:42597
I20260925 19:13:49.817059 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:42597 every 8 connection(s)
I20260925 19:13:49.817384 18046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:49.818341 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:49.820076 18114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:49.823926 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18046
I20260925 19:13:49.824025 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:49.824194 28345 external_mini_cluster.cc:1428] Setting key a452ec34733629c9d705a174955b9098
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:49.829984 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.820724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42597 (local address 127.0.0.1:40874)
0925 19:13:49.820867 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.820914 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:49.821421 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:49.821670 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:49.821672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:49.821918 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:49.822078 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.822082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.823032 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.823033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.823784 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.823786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.824475 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.824476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.824534 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.824992 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:49.824999 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:49.826426 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:49.828483 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.828489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.828491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.828674 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.828971 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.828972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.828972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.829062 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.829543 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:49.829549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:49.829607 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:13:49.829694 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:13:49.829819 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260925 19:13:49.830070 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.820791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.27.174.126:42597)
0925 19:13:49.820957 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:49.820961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:49.820990 (+ 29us) server_negotiation.cc:407] Connection header received
0925 19:13:49.821548 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:49.821549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:49.821565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:49.821637 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:49.822157 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.822947 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.823864 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.824390 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.824502 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.826506 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:49.826513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:49.826513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:49.826520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:49.828404 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.828757 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.828761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.828761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.828873 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.829149 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.829150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.829151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.829458 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:49.829555 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:49.829758 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 19:13:49.829906 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260925 19:13:49.830775 18114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:49.831080 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.830992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33946)
0925 19:13:49.831023 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.831062 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 19:13:49.831238 18065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
I20260925 19:13:49.831686 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.831558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33948)
0925 19:13:49.831636 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.831664 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:13:49.831853 18114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:44579: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
I20260925 19:13:49.850862 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:44579
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:49.881100 18114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } attempt: 1
I20260925 19:13:49.881670 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.881521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33952)
0925 19:13:49.881596 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.881638 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 19:13:49.881990 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.881898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33956)
0925 19:13:49.881943 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.881973 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:13:49.882081 18114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:44579: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
W20260925 19:13:49.931246 18121 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:49.931471 18121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:49.931499 18121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:49.931553 18121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:49.932871 18121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:49.932931 18121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:49.932955 18121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:49.932981 18121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:49.933012 18121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:49.934446 18121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:49.934713 18121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:49.934943 18121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:49.939833 18126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:49.939823 18129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:49.939847 18127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:49.939880 18121 server_base.cc:1034] running on GCE node
I20260925 19:13:49.940306 18121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:49.940584 18121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:49.941727 18121 hybrid_clock.cc:648] HybridClock initialized: now 1790363629941700 us; error 44 us; skew 500 ppm
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:49.944233 18121 init.cc:377] Logged in from keytab as kudu/127.27.174.125@KRBTEST.COM (short username kudu)
I20260925 19:13:49.944674 18121 webserver.cc:460] Webserver started at http://127.27.174.125:37767/ using document root <none> and password file <none>
I20260925 19:13:49.944926 18121 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:49.944968 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:49.945053 18121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:49.946031 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data/instance:
uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "60b0fbdaefea6a0fbf6fc44f30214723"
server_key_iv: "a04c771704dd889e24a41c1c2679646d"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.946322 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal/instance:
uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "60b0fbdaefea6a0fbf6fc44f30214723"
server_key_iv: "a04c771704dd889e24a41c1c2679646d"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.947527 18121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:13:49.948315 18136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:49.948488 18121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:13:49.948588 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d"
format_stamp: "Formatted at 2026-09-25 19:13:49 on dist-test-slave-fcdn"
server_key: "60b0fbdaefea6a0fbf6fc44f30214723"
server_key_iv: "a04c771704dd889e24a41c1c2679646d"
server_key_version: "encryptionkey@0"
I20260925 19:13:49.948683 18121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:49.950237 18114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } attempt: 2
I20260925 19:13:49.950644 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.950504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33968)
0925 19:13:49.950568 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.950607 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:13:49.951138 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.950927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33972)
0925 19:13:49.951074 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.951106 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260925 19:13:49.951339 18114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:44579: Network error: Client connection negotiation failed: client connection to 127.27.174.125:44579: connect: Connection refused (error 111)
I20260925 19:13:49.958921 18121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:49.961201 18121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:49.961359 18121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:49.965214 18121 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:44579
I20260925 19:13:49.965247 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:44579 every 8 connection(s)
I20260925 19:13:49.965579 18121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
I20260925 19:13:49.966365 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:49.967942 18189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:49.975237 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18121
I20260925 19:13:49.975317 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-1/wal/instance
I20260925 19:13:49.975513 28345 external_mini_cluster.cc:1428] Setting key 4a9ad1f0c5c040259545ee651a0b6d09
I20260925 19:13:49.976441 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.968584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.27.174.126:42597)
0925 19:13:49.968661 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:49.968664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:49.968852 (+ 188us) server_negotiation.cc:407] Connection header received
0925 19:13:49.969348 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:49.969349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:49.969363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:49.969426 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:49.970014 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.970516 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.971485 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.971772 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.971846 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.973813 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:49.973814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:49.973814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:49.973822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:49.975010 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.975452 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.975453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.975453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.975511 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.975805 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.975806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.975806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.975927 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:49.976004 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:49.976270 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 19:13:49.976354 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:13:49.976629 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.968557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42597 (local address 127.0.0.1:40886)
0925 19:13:49.968740 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.968773 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:49.969243 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:49.969495 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:49.969498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:49.969735 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:49.969915 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.969919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.970615 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.970616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.971379 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.971381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.971864 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.971866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.971985 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.972411 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:49.972419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:49.973704 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:49.975100 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.975107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.975109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.975391 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.975588 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.975589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.975589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.975733 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.975995 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:49.976001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:49.976140 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 19:13:49.976249 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:13:49.976397 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260925 19:13:49.977182 18189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:49 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.124@KRBTEST.COM; defaulting to no policy
I20260925 19:13:49.984196 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.977419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33982)
0925 19:13:49.977468 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.977479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:49.977522 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:49.977762 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:49.977763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:49.977904 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:49.977947 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.977948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.978773 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.978774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.979309 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:49.979311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.979852 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.979853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:49.979919 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.980331 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:49.980335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:49.981437 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:49.983201 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.983201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.983202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.983364 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.983625 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:49.983626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:49.983626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:49.983685 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:49.983974 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:49.983975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:49.984010 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 19:13:49.984061 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:13:49.984134 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:13:49.984361 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.977487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.27.174.125:44579)
0925 19:13:49.977635 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:49.977639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:49.977654 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:13:49.977670 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:49.977672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:49.977681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:49.977761 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:49.978021 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.978675 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.979404 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:49.979762 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:49.979853 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:49.981518 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:49.981526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:49.981526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:49.981532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:49.983107 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.983454 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.983457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.983457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.983513 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:49.983755 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:49.983762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:49.983763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:49.983905 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:49.983978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:49.984084 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 19:13:49.984207 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260925 19:13:49.984980 18189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:49.985276 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.985188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57786)
0925 19:13:49.985223 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.985261 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 19:13:49.985424 18138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.124@KRBTEST.COM" created.
I20260925 19:13:49.985905 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:49.985804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57798)
0925 19:13:49.985854 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:49.985882 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:13:49.986011 18189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:37589: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
I20260925 19:13:50.002533 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:37589
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:50.035251 18189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } attempt: 1
I20260925 19:13:50.035859 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.035710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57810)
0925 19:13:50.035772 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.035822 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:13:50.036247 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.036142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57812)
0925 19:13:50.036189 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.036221 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:13:50.036384 18189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:37589: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
I20260925 19:13:50.042538 18114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } attempt: 3
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:50.049732 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.042948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.27.174.125:44579)
0925 19:13:50.043008 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.043010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.043021 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:50.043085 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.043085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.043097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.043171 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.043513 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.044065 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.044770 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.044999 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.045080 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.047161 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.047162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.047162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.047168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.048663 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.049048 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.049049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.049049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.049096 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.049299 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.049299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.049300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.049402 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.049451 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.049606 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.049662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:13:50.049777 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.042889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33996)
0925 19:13:50.042934 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.042946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.043016 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.043179 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.043180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.043368 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.043431 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.043433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.044156 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.044157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.044689 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.044691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.045077 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.045079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.045156 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.045625 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.045630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.047075 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.048756 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.048757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.048758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.048971 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.049171 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.049172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.049172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.049239 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.049460 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.049461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.049511 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.049581 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.049684 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 19:13:50.050194 18114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:50.050508 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.050415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57824)
0925 19:13:50.050457 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.050486 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 19:13:50.050812 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.050758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57836)
0925 19:13:50.050779 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.050798 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 19:13:50.050908 18114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:37589: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
I20260925 19:13:50.082077 18114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } attempt: 1
I20260925 19:13:50.082624 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.082460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57842)
0925 19:13:50.082551 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.082593 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 19:13:50.083071 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.082983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57858)
0925 19:13:50.083025 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.083051 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:13:50.083186 18114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:37589: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
W20260925 19:13:50.086153 18196 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:50.086377 18196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:50.086432 18196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:50.086472 18196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:50.087714 18196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:50.087771 18196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:50.087796 18196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:50.087823 18196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:50.087849 18196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:50.089353 18196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42597,127.27.174.125:44579,127.27.174.124:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:50.089676 18196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:50.089897 18196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:50.094885 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:50.094962 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:50.094947 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:50.095330 18196 server_base.cc:1034] running on GCE node
I20260925 19:13:50.095543 18196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:50.095860 18196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:50.097041 18196 hybrid_clock.cc:648] HybridClock initialized: now 1790363630097025 us; error 27 us; skew 500 ppm
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:50.099712 18196 init.cc:377] Logged in from keytab as kudu/127.27.174.124@KRBTEST.COM (short username kudu)
I20260925 19:13:50.100138 18196 webserver.cc:460] Webserver started at http://127.27.174.124:41005/ using document root <none> and password file <none>
I20260925 19:13:50.100399 18196 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:50.100467 18196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:50.100610 18196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:50.101745 18196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data/instance:
uuid: "67e9ac454b1e487287c8d85d50a828c3"
format_stamp: "Formatted at 2026-09-25 19:13:50 on dist-test-slave-fcdn"
server_key: "029888a51590d449df741a1e882f5d3d"
server_key_iv: "b9562684182d227f08fd47d6217fbd82"
server_key_version: "encryptionkey@0"
I20260925 19:13:50.102094 18196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal/instance:
uuid: "67e9ac454b1e487287c8d85d50a828c3"
format_stamp: "Formatted at 2026-09-25 19:13:50 on dist-test-slave-fcdn"
server_key: "029888a51590d449df741a1e882f5d3d"
server_key_iv: "b9562684182d227f08fd47d6217fbd82"
server_key_version: "encryptionkey@0"
I20260925 19:13:50.103456 18196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:13:50.104292 18211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:50.104467 18196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:13:50.104545 18189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } attempt: 2
I20260925 19:13:50.104578 18196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "67e9ac454b1e487287c8d85d50a828c3"
format_stamp: "Formatted at 2026-09-25 19:13:50 on dist-test-slave-fcdn"
server_key: "029888a51590d449df741a1e882f5d3d"
server_key_iv: "b9562684182d227f08fd47d6217fbd82"
server_key_version: "encryptionkey@0"
I20260925 19:13:50.104667 18196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:50.104933 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.104823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57874)
0925 19:13:50.104868 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.104906 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:13:50.105317 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.105212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57880)
0925 19:13:50.105262 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.105297 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:13:50.105405 18189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:37589: Network error: Client connection negotiation failed: client connection to 127.27.174.124:37589: connect: Connection refused (error 111)
I20260925 19:13:50.119063 18196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:50.121423 18196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:50.121587 18196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:50.125553 18196 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:37589
I20260925 19:13:50.125602 18263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:37589 every 8 connection(s)
I20260925 19:13:50.125963 18196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
I20260925 19:13:50.126792 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:50.127297 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18196
I20260925 19:13:50.127388 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363366557183-28345-0/minicluster-data/master-2/wal/instance
I20260925 19:13:50.127552 28345 external_mini_cluster.cc:1428] Setting key 28b2a28f3fbafe63f55e3034a2057717
I20260925 19:13:50.128870 18264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:13:50.137902 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.129611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.27.174.126:42597)
0925 19:13:50.129697 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.129699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.129840 (+ 141us) server_negotiation.cc:407] Connection header received
0925 19:13:50.129895 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.129896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.129904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.129952 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.130553 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.131005 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.132183 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.132472 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.132535 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.134865 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.134867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.134867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.134874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.136456 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.136889 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.136890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.136890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.136947 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.137206 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.137206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.137207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.137314 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.137365 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.137714 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.137808 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 19:13:50.138262 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.128096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.27.174.124:37589)
0925 19:13:50.128282 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.128289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.128313 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:13:50.128709 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.128715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.128758 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.128869 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.129711 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.130865 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.131976 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.132357 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.132461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.135196 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.135201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.135201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.135207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.137106 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.137468 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.137471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.137471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.137516 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.137722 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.137723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.137724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.137880 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.138021 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.138137 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.138193 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260925 19:13:50.138268 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.129563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42597 (local address 127.0.0.1:40898)
0925 19:13:50.129732 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.129767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.129842 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.129993 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.129995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.130332 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.130445 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.130451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.131092 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.131093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.132103 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.132106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.132559 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.132560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.132636 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.133062 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.133072 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.134753 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.136557 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.136563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.136565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.136810 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.137018 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.137018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.137019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.137112 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.137380 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.137387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.137532 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.137686 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.137830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260925 19:13:50.138847 18264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:50.146589 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.139184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.27.174.125:44579)
0925 19:13:50.139242 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.139244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.139548 (+ 304us) server_negotiation.cc:407] Connection header received
0925 19:13:50.139659 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.139660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.139671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.139725 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.139993 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.140464 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.141117 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.141335 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.141420 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.143718 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.143719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.143720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.143726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.145427 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.145814 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.145815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.145815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.145866 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.146070 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.146071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.146072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.146211 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.146284 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.146418 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.146490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:13:50.146600 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.139116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:44579 (local address 127.0.0.1:33998)
0925 19:13:50.139441 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.139449 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.139558 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.139729 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.139730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.139866 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.139911 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.139912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.140546 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.140547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.141033 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.141035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.141420 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.141420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.141489 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.141917 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.141922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.143581 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.145507 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.145507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.145508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.145736 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.145932 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.145933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.145933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.146002 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.146284 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.146285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.146320 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.146372 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.146489 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314}
I20260925 19:13:50.146797 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.139405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.27.174.126:42597)
0925 19:13:50.139540 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.139542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.139550 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:50.139567 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.139568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.139576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.139707 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.140177 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.140635 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.141310 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.141506 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.141590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.143781 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.143782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.143783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.143789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.145586 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.145986 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.145987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.145987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.146051 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.146232 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.146233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.146234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.146355 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.146549 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.146585 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.146663 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 19:13:50.147336 18264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:50.154757 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.147641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57902)
0925 19:13:50.147679 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.147687 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.147719 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.147849 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.147850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.147955 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.147982 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.147983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.148699 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.148699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.149236 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.149238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.149690 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.149691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.149748 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.150244 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.150248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.151746 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.153673 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.153674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.153674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.153909 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.154110 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.154110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.154111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.154174 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.154526 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.154526 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.154574 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.154626 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.154692 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 19:13:50.155015 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.147684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.27.174.124:37589)
0925 19:13:50.147717 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.147718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.147725 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:13:50.147768 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.147768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.147777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.147838 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.148050 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.148630 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.149328 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.149614 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.149664 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.151853 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.151855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.151855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.151863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.153581 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.153993 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.153994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.153994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.154044 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.154300 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.154301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.154301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.154411 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.154533 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.154674 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.154750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:13:50.156469 18264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: Bootstrap starting.
I20260925 19:13:50.157275 18264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:50.157505 18264 log.cc:824] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:50.157923 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.147439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.27.174.125:44579)
0925 19:13:50.147492 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.147494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.147501 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:13:50.147588 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.147589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.147620 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.147742 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.148321 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.148804 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.149492 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.149742 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.149836 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.152674 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.152675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.152675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.152681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.154420 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.154957 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.154958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.154958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.155005 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.155236 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.155237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.155238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.155355 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.157767 (+ 2412us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.157804 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.157865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:13:50.158270 18264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: No bootstrap required, opened a new log
I20260925 19:13:50.158326 18114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } attempt: 2
I20260925 19:13:50.159373 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:50.160408 18264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.160557 18264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9ac454b1e487287c8d85d50a828c3, State: Initialized, Role: FOLLOWER
I20260925 19:13:50.160627 18264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.160888 18270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.160959 18264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:50.161057 18270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.160377 18088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2f686039001940e9a6826ee538f4e31f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
W20260925 19:13:50.163272 18281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:50.163622 18281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:50.163301 18196 master_runner.cc:186] Error getting master registration for 127.27.174.126:42597: OK, instance_id { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" instance_seqno: 1790363629815833 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:13:50.166154 18196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:13:50.166249 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.158563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57918)
0925 19:13:50.158613 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.158623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.158657 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.158832 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.158833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.158995 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.159036 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.159038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.159902 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.159904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.160674 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.160677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.161140 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.161141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.161202 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.161638 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.161644 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.163174 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.165053 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.165055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.165056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.165289 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.165530 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.165531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.165531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.165606 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.165905 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.165906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.165968 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.166041 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.166147 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:13:50.166265 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.158593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.27.174.124:37589)
0925 19:13:50.158743 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.158745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.158752 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:13:50.158771 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.158772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.158780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.158826 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.159122 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.159811 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.160776 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.161040 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.161365 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.163265 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.163266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.163266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.163272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.164950 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.165386 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.165387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.165388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.165462 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.165686 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.165686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.165687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.165835 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.165908 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.166080 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.166153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260925 19:13:50.167694 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f: Bootstrap starting.
I20260925 19:13:50.168375 18114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:50.168677 18114 log.cc:824] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:50.169375 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f: No bootstrap required, opened a new log
I20260925 19:13:50.170338 18114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.170465 18114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f686039001940e9a6826ee538f4e31f, State: Initialized, Role: FOLLOWER
I20260925 19:13:50.170550 18114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.170890 18114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:50.170909 18088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:13:50.170953 18088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:50.170861 18284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.171828 18088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.172116 18088 leader_election.cc:288] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [CANDIDATE]: Term 1 election: Requested vote from peers 24ab7141d1364bc5b8cf8bce4b06dc9d (127.27.174.125:44579), 67e9ac454b1e487287c8d85d50a828c3 (127.27.174.124:37589)
I20260925 19:13:50.171833 18284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.173393 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f686039001940e9a6826ee538f4e31f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67e9ac454b1e487287c8d85d50a828c3"
I20260925 19:13:50.173517 18238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:50.174240 18238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f686039001940e9a6826ee538f4e31f in term 1.
I20260925 19:13:50.174477 18063 leader_election.cc:302] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f686039001940e9a6826ee538f4e31f, 67e9ac454b1e487287c8d85d50a828c3; no voters:
I20260925 19:13:50.174604 18284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:50.174772 18284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [term 1 LEADER]: Becoming Leader. State: Replica: 2f686039001940e9a6826ee538f4e31f, State: Running, Role: LEADER
I20260925 19:13:50.174847 18284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.175343 18300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f686039001940e9a6826ee538f4e31f. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.175431 18300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:50.175628 18302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:50.176067 18302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:50.173466 18046 master_runner.cc:186] Error getting master registration for 127.27.174.125:44579: OK, instance_id { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" instance_seqno: 1790363629964117 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:13:50.176450 18046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:13:50.178920 18238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f686039001940e9a6826ee538f4e31f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:50.179224 18300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:50.180521 18270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f686039001940e9a6826ee538f4e31f. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.180608 18270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.181888 18284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.181983 18284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:50.182086 18270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.182133 18304 mvcc.cc:205] Tried to move back new op lower bound from 7333329429206056960 to 7333329429196877824. Current Snapshot: MvccSnapshot[applied={T|T < 7333329429206056960}]
I20260925 19:13:50.182163 18270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.183079 18308 mvcc.cc:205] Tried to move back new op lower bound from 7333329429206056960 to 7333329429196877824. Current Snapshot: MvccSnapshot[applied={T|T < 7333329429206056960}]
I20260925 19:13:50.185989 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.173448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.27.174.125:44579)
0925 19:13:50.173492 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.173494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.177533 (+ 4039us) server_negotiation.cc:407] Connection header received
0925 19:13:50.177642 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.177643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.177654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.177684 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.178750 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.179503 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.180738 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.181049 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.181090 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.182881 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.182883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.182883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.182890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.184779 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.185155 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.185156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.185156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.185204 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.185412 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.185413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.185413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.185532 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.185601 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.185799 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.185872 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:13:50.192106 18302 catalog_manager.cc:1347] Generated new cluster ID: b309d90253f347ecabde1031a61172d4
I20260925 19:13:50.192178 18302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:50.195125 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.173597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.27.174.126:42597)
0925 19:13:50.173646 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.173649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.183449 (+ 9800us) server_negotiation.cc:407] Connection header received
0925 19:13:50.184061 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.184063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.184073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.184102 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.187203 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.187947 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.188949 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.189267 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.189333 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.190779 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.190780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.190781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.190788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.192629 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.193068 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.193069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.193070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.193122 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.193329 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.193330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.193330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.193450 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.193517 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.193987 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.194065 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:13:50.196556 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.174545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.27.174.124:37589)
0925 19:13:50.174578 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.174580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.183917 (+ 9337us) server_negotiation.cc:407] Connection header received
0925 19:13:50.187190 (+ 3273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.187191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.187203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.187260 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.188559 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.189286 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.190961 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.191253 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.191294 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.193385 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.193386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.193387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.193393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.194953 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.195501 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.195502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.195503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.195552 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.195928 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.195929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.195929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.196051 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.196121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.196371 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.196451 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:50.196630 18189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } attempt: 3
Sep 25 19:13:50 dist-test-slave-fcdn krb5kdc[18033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:13:50.205641 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.196942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:37589 (local address 127.0.0.1:57928)
0925 19:13:50.197081 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:50.197091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:50.197138 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:50.197269 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:50.197269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:50.197426 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:50.197468 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.197470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.198782 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.198783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.199535 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:50.199537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.200024 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.200025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:50.200076 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.200468 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:50.200473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:50.201904 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:50.204406 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.204407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.204408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.204625 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.204911 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:50.204912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:50.204912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:50.204981 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:50.205345 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:50.205346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:50.205397 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:13:50.205443 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:13:50.205537 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260925 19:13:50.206353 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:50.196975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.27.174.124:37589)
0925 19:13:50.197006 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.197008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.197125 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:13:50.197191 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.197191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.197200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.197256 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.197549 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.198257 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.199629 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:50.199927 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:50.200022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:50.201979 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:50.201981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:50.201981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:50.201988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:50.204268 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.204720 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.204721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.204721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.204820 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:50.205060 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:50.205061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:50.205061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:50.205245 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:50.206059 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:50.206100 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:13:50.206218 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 19:13:50.208150 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: Bootstrap starting.
I20260925 19:13:50.208874 18189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:50.209187 18189 log.cc:824] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:50.224407 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: No bootstrap required, opened a new log
I20260925 19:13:50.226485 18189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.226626 18189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ab7141d1364bc5b8cf8bce4b06dc9d, State: Initialized, Role: FOLLOWER
I20260925 19:13:50.226703 18189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } }
I20260925 19:13:50.227034 18189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:50.227945 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:50.228881 18162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [term 1 FOLLOWER]: Refusing update from remote peer 2f686039001940e9a6826ee538f4e31f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:50.229280 18300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:50.229372 18163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f686039001940e9a6826ee538f4e31f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d"
I20260925 19:13:50.230202 18163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f686039001940e9a6826ee538f4e31f in term 1.
I20260925 19:13:50.231721 18313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.231853 18313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.231913 18313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f686039001940e9a6826ee538f4e31f. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.231966 18313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.232003 18313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.232057 18313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:50.233184 18318 mvcc.cc:205] Tried to move back new op lower bound from 7333329429206056960 to 7333329429196877824. Current Snapshot: MvccSnapshot[applied={T|T < 7333329429206056960}]
W20260925 19:13:50.234489 18328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:50.234618 18328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:50.266160 18300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f686039001940e9a6826ee538f4e31f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f686039001940e9a6826ee538f4e31f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42597 } } peers { permanent_uuid: "24ab7141d1364bc5b8cf8bce4b06dc9d" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 44579 } } peers { permanent_uuid: "67e9ac454b1e487287c8d85d50a828c3" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 37589 } } }
I20260925 19:13:50.266287 18300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:50.267534 18302 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:50.268249 18302 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:13:50.279433 18302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f686039001940e9a6826ee538f4e31f: Generated new TSK 0
I20260925 19:13:50.279647 18302 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:13:50.304181 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.302748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.27.174.125:44579)
0925 19:13:50.302805 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.302807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.303452 (+ 645us) server_negotiation.cc:407] Connection header received
0925 19:13:50.303566 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.303567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.303579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.303638 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.304105 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34026: BlockingRecv error: recv got EOF from 127.0.0.1:34026 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 19:13:50.304947 18267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.303180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.27.174.124:37589)
0925 19:13:50.303218 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.303221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.304441 (+ 1220us) server_negotiation.cc:407] Connection header received
0925 19:13:50.304538 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.304538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.304550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.304579 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.304890 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57944: BlockingRecv error: recv got EOF from 127.0.0.1:57944 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:13:50.305945 18117 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.305274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.27.174.126:42597)
0925 19:13:50.305312 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.305314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.305433 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:13:50.305547 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.305549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.305560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.305590 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.305888 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40918: BlockingRecv error: recv got EOF from 127.0.0.1:40918 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:13:50.308367 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.307567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.27.174.125:44579)
0925 19:13:50.307623 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.307625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.307813 (+ 188us) server_negotiation.cc:407] Connection header received
0925 19:13:50.307920 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.307922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.307941 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.307971 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.308324 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34040: BlockingRecv error: recv got EOF from 127.0.0.1:34040 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:13:50.311373 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.310600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.27.174.125:44579)
0925 19:13:50.310632 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.310633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.310903 (+ 270us) server_negotiation.cc:407] Connection header received
0925 19:13:50.310999 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.311000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.311009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.311037 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.311332 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34042: BlockingRecv error: recv got EOF from 127.0.0.1:34042 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:13:50.315094 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.314343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.27.174.125:44579)
0925 19:13:50.314374 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.314376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.314603 (+ 227us) server_negotiation.cc:407] Connection header received
0925 19:13:50.314697 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.314699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.314708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.314737 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.315055 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34052: BlockingRecv error: recv got EOF from 127.0.0.1:34052 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 19:13:50.320892 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.320157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.27.174.125:44579)
0925 19:13:50.320188 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.320190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.320413 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:13:50.320506 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.320506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.320515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.320543 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.320853 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34058: BlockingRecv error: recv got EOF from 127.0.0.1:34058 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:13:50.330857 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.329994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.27.174.125:44579)
0925 19:13:50.330031 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.330033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.330294 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:13:50.330398 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.330399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.330410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.330441 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.330805 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34070: BlockingRecv error: recv got EOF from 127.0.0.1:34070 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:13:50.349145 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.348226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.27.174.125:44579)
0925 19:13:50.348270 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.348272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.348529 (+ 257us) server_negotiation.cc:407] Connection header received
0925 19:13:50.348636 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.348637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.348649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.348678 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.349071 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34080: BlockingRecv error: recv got EOF from 127.0.0.1:34080 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:13:50.383342 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.382570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.27.174.125:44579)
0925 19:13:50.382622 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.382624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.382765 (+ 141us) server_negotiation.cc:407] Connection header received
0925 19:13:50.382803 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.382805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.382817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.382895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.383265 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34094: BlockingRecv error: recv got EOF from 127.0.0.1:34094 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 19:13:50.449229 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.448521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.27.174.125:44579)
0925 19:13:50.448609 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.448611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.448751 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:13:50.448784 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.448785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.448796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.448842 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.449153 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34096: BlockingRecv error: recv got EOF from 127.0.0.1:34096 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 19:13:50.579035 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.578354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.27.174.125:44579)
0925 19:13:50.578427 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.578429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.578477 (+ 48us) server_negotiation.cc:407] Connection header received
0925 19:13:50.578539 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.578540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.578550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.578604 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.578950 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34100: BlockingRecv error: recv got EOF from 127.0.0.1:34100 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 19:13:50.837054 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:50.836329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.27.174.125:44579)
0925 19:13:50.836396 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:50.836398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:50.836420 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:13:50.836498 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:50.836499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:50.836508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:50.836553 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:50.836986 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34116: BlockingRecv error: recv got EOF from 127.0.0.1:34116 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:13:51.164186 18281 catalog_manager.cc:1259] Loaded cluster ID: b309d90253f347ecabde1031a61172d4
I20260925 19:13:51.164294 18281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: loading cluster ID for follower catalog manager: success
I20260925 19:13:51.165097 18281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: acquiring CA information for follower catalog manager: success
I20260925 19:13:51.165409 18281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67e9ac454b1e487287c8d85d50a828c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:13:51.235126 18328 catalog_manager.cc:1259] Loaded cluster ID: b309d90253f347ecabde1031a61172d4
I20260925 19:13:51.235213 18328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: loading cluster ID for follower catalog manager: success
I20260925 19:13:51.236095 18328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: acquiring CA information for follower catalog manager: success
I20260925 19:13:51.236392 18328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24ab7141d1364bc5b8cf8bce4b06dc9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:13:51.352792 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:51.350141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.27.174.125:44579)
0925 19:13:51.350241 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:51.350243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:51.350269 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:13:51.350345 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:51.350346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:51.350394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:51.350467 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:51.350630 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.351139 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.351980 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.352188 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.352258 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:51.352425 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:51.352567 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:51.352641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:51.352685 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 19:13:51.352724 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
I20260925 19:13:51.357101 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:51.354656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.27.174.124:37589)
0925 19:13:51.354760 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:51.354762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:51.354771 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:51.354836 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:51.354837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:51.354880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:51.354973 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:51.355063 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.355541 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.356325 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.356541 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.356616 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:51.356752 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:51.356865 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:51.356951 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:51.356992 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:13:51.357011 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260925 19:13:51.361560 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:51.358869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.27.174.125:44579)
0925 19:13:51.358907 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:51.358909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:51.358917 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:51.358943 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:51.358944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:51.358952 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:51.359015 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:51.359122 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.359822 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.360703 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.361011 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.361073 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:51.361254 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:51.361288 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:51.361336 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:51.361431 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:13:51.361459 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:13:51.361940 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:51.358882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.27.174.124:37589)
0925 19:13:51.358914 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:51.358916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:51.359315 (+ 399us) server_negotiation.cc:407] Connection header received
0925 19:13:51.359337 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:51.359338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:51.359347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:51.359393 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:51.359524 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.360249 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.361076 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.361409 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.361469 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:51.361607 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:51.361638 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:51.361691 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:51.361835 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 19:13:51.361855 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:13:51.362331 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:51.358771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.27.174.126:42597)
0925 19:13:51.358881 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:51.358883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:51.359166 (+ 283us) server_negotiation.cc:407] Connection header received
0925 19:13:51.359227 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:51.359228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:51.359261 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:51.359351 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:51.359558 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.360317 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.361293 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:51.361575 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:51.361629 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:51.361909 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:51.362007 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:51.362067 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:51.362233 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 19:13:51.362256 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260925 19:13:51.364120 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18046
I20260925 19:13:51.369541 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18121
I20260925 19:13:51.374475 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18196
2026-09-25T19:13:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2828 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:13:51 dist-test-slave-fcdn krb5kdc[18412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:13:51 dist-test-slave-fcdn krb5kdc[18412](info): set up 2 sockets
Sep 25 19:13:51 dist-test-slave-fcdn krb5kdc[18412](info): commencing operation
krb5kdc: starting...
W20260925 19:13:52.408020 28345 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/test-user.keytab.
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20260925 19:13:52.527063 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41877
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:13:52.605960 18425 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:52.606200 18425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:52.606230 18425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:52.606299 18425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:52.607450 18425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:52.607512 18425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:52.607534 18425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:52.607554 18425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:52.607583 18425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:52.609069 18425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:52.609338 18425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:52.609558 18425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:52.614291 18431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.614322 18433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.614436 18430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:52.614331 18425 server_base.cc:1034] running on GCE node
I20260925 19:13:52.614696 18425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:52.614943 18425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:52.616142 18425 hybrid_clock.cc:648] HybridClock initialized: now 1790363632616111 us; error 43 us; skew 500 ppm
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:52.635494 18425 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20260925 19:13:52.636023 18425 webserver.cc:460] Webserver started at http://127.27.174.126:39477/ using document root <none> and password file <none>
I20260925 19:13:52.636277 18425 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:52.636325 18425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:52.636480 18425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:52.637576 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data/instance:
uuid: "bae91d5714f64ac294d4dcfff21bfb83"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "302522c31e5539088107c3a4ff44802f"
server_key_iv: "8ee5301b27d6759e8753d1d02ff5bc82"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.637950 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance:
uuid: "bae91d5714f64ac294d4dcfff21bfb83"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "302522c31e5539088107c3a4ff44802f"
server_key_iv: "8ee5301b27d6759e8753d1d02ff5bc82"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.639400 18425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:52.640256 18440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:52.640434 18425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:52.640537 18425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal
uuid: "bae91d5714f64ac294d4dcfff21bfb83"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "302522c31e5539088107c3a4ff44802f"
server_key_iv: "8ee5301b27d6759e8753d1d02ff5bc82"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.640631 18425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:52.658814 18425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:52.660727 18425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:52.660928 18425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:52.664944 18425 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41877
I20260925 19:13:52.664980 18492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41877 every 8 connection(s)
I20260925 19:13:52.665324 18425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/data/info.pb
I20260925 19:13:52.666311 18493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:52.667982 18493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:52.672165 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18425
I20260925 19:13:52.672281 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-0/wal/instance
I20260925 19:13:52.672518 28345 external_mini_cluster.cc:1428] Setting key 1a0f08e9347f1322ab2de98ed56eaa05
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:52.677508 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.668693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.27.174.126:41877)
0925 19:13:52.668871 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.668875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.668930 (+ 55us) server_negotiation.cc:407] Connection header received
0925 19:13:52.669436 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.669438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.669455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.669539 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.670066 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.670859 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.671707 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.672032 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.672183 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.674305 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.674310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.674311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.674319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.675946 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.676333 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.676335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.676336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.676416 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.676658 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.676659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.676659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.676860 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.676951 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.677143 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.677271 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260925 19:13:52.677513 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.668660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41877 (local address 127.0.0.1:51914)
0925 19:13:52.668816 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.668856 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.669308 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.669565 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.669567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.669810 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.669953 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.669956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.670942 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.670942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.671617 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.671619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.672143 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.672144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.672208 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.672774 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.672782 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.674207 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.676042 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.676048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.676050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.676238 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.676484 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.676484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.676485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.676574 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.676930 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.676935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.676993 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.677104 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.677319 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":88,"thread_start_us":77,"threads_started":1}
I20260925 19:13:52.678217 18493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:52.678547 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.678445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37824)
0925 19:13:52.678490 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.678528 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:13:52.678717 18443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
I20260925 19:13:52.679174 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.679047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37830)
0925 19:13:52.679091 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.679158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:13:52.679286 18493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:39009: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
I20260925 19:13:52.698467 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:39009
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:52.728534 18493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } attempt: 1
I20260925 19:13:52.729187 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.729024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37836)
0925 19:13:52.729099 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.729144 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 19:13:52.729630 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.729541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37848)
0925 19:13:52.729585 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.729611 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:13:52.729799 18493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:39009: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
W20260925 19:13:52.779798 18500 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:52.779994 18500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:52.780011 18500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:52.780054 18500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:52.781213 18500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:52.781258 18500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:52.781272 18500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:52.781322 18500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:52.781343 18500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:52.782785 18500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:52.783032 18500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:52.783275 18500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:52.788174 18506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.788213 18505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.788216 18508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:52.788362 18500 server_base.cc:1034] running on GCE node
I20260925 19:13:52.788671 18500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:52.788949 18500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:52.790113 18500 hybrid_clock.cc:648] HybridClock initialized: now 1790363632790115 us; error 43 us; skew 500 ppm
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:52.792577 18500 init.cc:377] Logged in from keytab as kudu/127.27.174.125@KRBTEST.COM (short username kudu)
I20260925 19:13:52.793051 18500 webserver.cc:460] Webserver started at http://127.27.174.125:46143/ using document root <none> and password file <none>
I20260925 19:13:52.793277 18500 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:52.793346 18500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:52.793491 18500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:52.794589 18500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data/instance:
uuid: "0e7272354b9d41338aa22cffd25ac9d7"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "9c8dea6e261c8817a6aca9dd769f2347"
server_key_iv: "f5691cce111b806633bf0b76fe346b08"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.794942 18500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal/instance:
uuid: "0e7272354b9d41338aa22cffd25ac9d7"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "9c8dea6e261c8817a6aca9dd769f2347"
server_key_iv: "f5691cce111b806633bf0b76fe346b08"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.796365 18500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:13:52.797142 18515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:52.797343 18500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:13:52.797441 18500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal
uuid: "0e7272354b9d41338aa22cffd25ac9d7"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "9c8dea6e261c8817a6aca9dd769f2347"
server_key_iv: "f5691cce111b806633bf0b76fe346b08"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.797519 18500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:52.797969 18493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } attempt: 2
I20260925 19:13:52.798334 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.798214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37860)
0925 19:13:52.798269 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.798307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:13:52.798627 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.798536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37862)
0925 19:13:52.798582 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.798605 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:13:52.798717 18493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.125:39009: Network error: Client connection negotiation failed: client connection to 127.27.174.125:39009: connect: Connection refused (error 111)
I20260925 19:13:52.809973 18500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:52.811810 18500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:52.811959 18500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:52.815824 18500 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:39009
I20260925 19:13:52.815852 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:39009 every 8 connection(s)
I20260925 19:13:52.816192 18500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/data/info.pb
I20260925 19:13:52.817024 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:52.818521 18568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:52.823071 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18500
I20260925 19:13:52.823175 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-1/wal/instance
I20260925 19:13:52.823365 28345 external_mini_cluster.cc:1428] Setting key b6a7c0440c36a23d8c8683f75cb5096d
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:52.827245 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.819165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.27.174.126:41877)
0925 19:13:52.819232 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.819234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.819416 (+ 182us) server_negotiation.cc:407] Connection header received
0925 19:13:52.819940 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.819941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.819953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.820005 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.820598 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.821080 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.821971 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.822159 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.822209 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.824480 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.824481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.824482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.824490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.825871 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.826305 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.826306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.826307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.826365 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.826612 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.826613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.826613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.826726 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.826802 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.827065 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.827145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:13:52.827428 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.819127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41877 (local address 127.0.0.1:51922)
0925 19:13:52.819293 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.819328 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.819853 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.820075 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.820078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.820345 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.820528 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.820532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.821157 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.821158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.821902 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.821903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.822229 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.822230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.822326 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.822834 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.822842 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.824380 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.825992 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.825999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.826001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.826215 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.826433 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.826434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.826434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.826544 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.826794 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.826800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.826937 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.827045 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.827196 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260925 19:13:52.828011 18568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.124@KRBTEST.COM" created.
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:52.838089 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.828328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.27.174.125:39009)
0925 19:13:52.828466 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.828471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.828510 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:13:52.828535 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.828537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.828553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.828647 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.828880 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.830013 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.830650 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.831126 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.831226 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.834993 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.835001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.835001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.835008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.836826 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.837225 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.837227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.837228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.837287 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.837492 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.837492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.837492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.837620 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.837681 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.837841 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.837981 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 19:13:52.838101 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.828291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37866)
0925 19:13:52.828322 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.828331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.828364 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.828645 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.828646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.828760 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.828809 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.828810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.830098 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.830099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.830567 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.830568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.831222 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.831223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.831275 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.831670 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.831675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.834896 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.836929 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.836930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.836930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.837143 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.837359 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.837360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.837360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.837430 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.837686 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.837687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.837735 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.837806 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.837916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 19:13:52.838807 18568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:52.839170 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.839032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53174)
0925 19:13:52.839086 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.839140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:13:52.839331 18519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
I20260925 19:13:52.839813 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.839694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53186)
0925 19:13:52.839749 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.839782 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:13:52.839931 18568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:38295: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
I20260925 19:13:52.850987 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwy_z0T/build/release/bin/kudu
/tmp/dist-test-taskwy_z0T/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:38295
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:13:52.889180 18568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } attempt: 1
I20260925 19:13:52.889714 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.889553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53198)
0925 19:13:52.889627 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.889676 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 19:13:52.889832 18493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } attempt: 3
I20260925 19:13:52.890053 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.889966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53206)
0925 19:13:52.889997 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.890031 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 19:13:52.890173 18568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:38295: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:52.897066 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.890145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.27.174.125:39009)
0925 19:13:52.890187 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.890189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.890200 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:13:52.890310 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.890311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.890323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.890374 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.890769 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.891247 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.892279 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.892488 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.892541 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.894449 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.894451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.894451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.894457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.896003 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.896411 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.896412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.896412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.896459 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.896649 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.896650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.896650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.896752 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.896800 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.896952 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.897009 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:13:52.897122 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.890106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37878)
0925 19:13:52.890158 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.890168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.890217 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.890413 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.890414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.890596 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.890673 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.890675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.891327 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.891328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.892201 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.892205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.892560 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.892561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.892637 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.893050 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.893056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.894353 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.896079 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.896080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.896080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.896331 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.896521 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.896522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.896522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.896591 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.896812 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.896812 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.896861 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.896929 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.897021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:13:52.897677 18493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:13:52.898038 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.897928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53212)
0925 19:13:52.897973 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.898007 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:13:52.898352 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.898262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53220)
0925 19:13:52.898310 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.898335 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:13:52.898444 18493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:38295: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
I20260925 19:13:52.929596 18493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } attempt: 1
I20260925 19:13:52.930060 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.929889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53232)
0925 19:13:52.929959 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.930008 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:13:52.930449 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.930343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53246)
0925 19:13:52.930393 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.930424 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:13:52.930553 18493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:38295: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
W20260925 19:13:52.934432 18575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:13:52.934633 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:13:52.934677 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:13:52.934715 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:13:52.935930 18575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:13:52.935968 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:13:52.935983 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:13:52.936015 18575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:13:52.936043 18575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:13:52.937489 18575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41877,127.27.174.125:39009,127.27.174.124:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:13:52.937740 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:13:52.937963 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:13:52.942760 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.942778 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:13:52.942744 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:13:52.942961 18575 server_base.cc:1034] running on GCE node
I20260925 19:13:52.943239 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:13:52.943521 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:13:52.944682 18575 hybrid_clock.cc:648] HybridClock initialized: now 1790363632944664 us; error 81 us; skew 500 ppm
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:13:52.947154 18575 init.cc:377] Logged in from keytab as kudu/127.27.174.124@KRBTEST.COM (short username kudu)
I20260925 19:13:52.947527 18575 webserver.cc:460] Webserver started at http://127.27.174.124:36695/ using document root <none> and password file <none>
I20260925 19:13:52.947788 18575 fs_manager.cc:359] Metadata directory not provided
I20260925 19:13:52.947834 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:13:52.947917 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:13:52.949046 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data/instance:
uuid: "21a47590e8d74ed0acbc113f62e638aa"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "53741bfca9a309d4838aace36b9744fc"
server_key_iv: "8f0a4f37c44d5dc865fda1af084a936e"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.949476 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal/instance:
uuid: "21a47590e8d74ed0acbc113f62e638aa"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "53741bfca9a309d4838aace36b9744fc"
server_key_iv: "8f0a4f37c44d5dc865fda1af084a936e"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.950819 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:13:52.951539 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:13:52.951718 18575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:13:52.951821 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal
uuid: "21a47590e8d74ed0acbc113f62e638aa"
format_stamp: "Formatted at 2026-09-25 19:13:52 on dist-test-slave-fcdn"
server_key: "53741bfca9a309d4838aace36b9744fc"
server_key_iv: "8f0a4f37c44d5dc865fda1af084a936e"
server_key_version: "encryptionkey@0"
I20260925 19:13:52.951906 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:13:52.958307 18568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } attempt: 2
I20260925 19:13:52.958712 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.958585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53262)
0925 19:13:52.958638 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.958676 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 19:13:52.959079 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.958986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53278)
0925 19:13:52.959030 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.959058 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:13:52.959203 18568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.174.124:38295: Network error: Client connection negotiation failed: client connection to 127.27.174.124:38295: connect: Connection refused (error 111)
I20260925 19:13:52.959949 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:13:52.962154 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:13:52.962312 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:13:52.966323 18575 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:38295
I20260925 19:13:52.966387 18642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:38295 every 8 connection(s)
I20260925 19:13:52.966711 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/data/info.pb
I20260925 19:13:52.967581 18643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:13:52.969189 18643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20260925 19:13:52.975646 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwy_z0T/build/release/bin/kudu as pid 18575
I20260925 19:13:52.975749 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwy_z0T/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363366557183-28345-0/minicluster-data/master-2/wal/instance
I20260925 19:13:52.975968 28345 external_mini_cluster.cc:1428] Setting key 795e31d6838923fea9a086c941bd6ed6
I20260925 19:13:52.978557 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.969882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.27.174.126:41877)
0925 19:13:52.969968 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.969971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.970172 (+ 201us) server_negotiation.cc:407] Connection header received
0925 19:13:52.970699 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.970701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.970713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.970779 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.971382 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.971955 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.972918 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.973142 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.973489 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.975488 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.975490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.975491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.975498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.977152 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.977596 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.977597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.977597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.977659 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.977962 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.977963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.977963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.978094 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.978164 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.978430 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.978495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:13:52.978932 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.969843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41877 (local address 127.0.0.1:51932)
0925 19:13:52.970022 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.970055 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.970589 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.970850 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.970853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.971095 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.971286 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.971290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.972061 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.972062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.972821 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.972823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.973488 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.973489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.973652 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.974096 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.974104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.975390 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.977241 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.977248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.977250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.977511 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.977758 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.977759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.977759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.977889 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.978156 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.978158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.978267 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.978406 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.978571 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260925 19:13:52.979569 18643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:52.985312 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.976364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.27.174.124:38295)
0925 19:13:52.976549 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.976553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.976573 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:13:52.976593 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.976595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.976610 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.976713 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.977341 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.978292 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.979333 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.979813 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.979879 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.982370 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.982376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.982376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.982382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.984073 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.984446 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.984449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.984450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.984510 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.984719 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.984720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.984720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.984855 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.985027 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.985074 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.985152 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260925 19:13:52.986956 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.979870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:39009 (local address 127.0.0.1:37890)
0925 19:13:52.979919 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.979929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.979971 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.980154 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.980155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.980302 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.980372 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.980374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.980977 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.980978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.981621 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.981624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.982091 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.982092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.982160 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.982532 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.982536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.983968 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.985897 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.985898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.985899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.986118 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.986315 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.986316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.986316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.986381 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.986648 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.986648 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.986700 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.986762 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.986866 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:13:52.986981 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.979958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.27.174.125:39009)
0925 19:13:52.980001 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.980003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.980010 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:13:52.980032 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.980033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.980045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.980092 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.980457 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.980900 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.981700 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.982013 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.982093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.984053 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.984055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.984055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.984061 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.985830 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.986187 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.986188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.986188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.986249 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.986447 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.986447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.986448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.986561 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.986672 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.986785 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.986882 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:13:52.987295 18643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:13:52.993696 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.986232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.27.174.126:41877)
0925 19:13:52.986302 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.986305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.986314 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:52.986336 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.986337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.986346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.986395 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.986828 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.987605 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.988285 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.988489 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.988573 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.990894 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.990895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.990896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.990903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.992577 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.992934 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.992934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.992935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.992981 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.993184 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.993185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.993185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.993317 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.993496 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.993534 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.993590 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:13:52.993956 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.987551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53288)
0925 19:13:52.987627 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:52.987637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:52.987677 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:52.987802 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:52.987803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:52.987929 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:52.987952 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.987953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.988567 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.988568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.989034 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:52.989036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.989406 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.989407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:52.989453 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.989816 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:52.989820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:52.991210 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:52.992873 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.992874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.992875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.993085 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.993332 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:52.993333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:52.993333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:52.993386 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:52.993662 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:52.993663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:52.993705 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 19:13:52.993777 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:13:52.993865 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:13:52.994239 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.987623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.27.174.124:38295)
0925 19:13:52.987662 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.987664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.987671 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:13:52.987732 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.987732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.987743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.987799 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.988031 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.988478 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.989117 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.989325 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.989407 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.991311 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.991312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.991313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.991319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:52.992795 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.993175 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.993176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.993176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.993242 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:52.993448 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:52.993449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:52.993449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:52.993577 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:52.993653 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:52.994059 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 19:13:52.994122 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:13:52.995890 18643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: Bootstrap starting.
I20260925 19:13:52.996897 18643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:52.997195 18643 log.cc:824] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:52.997922 18643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: No bootstrap required, opened a new log
Sep 25 19:13:52 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20260925 19:13:53.000018 18643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.000186 18643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a47590e8d74ed0acbc113f62e638aa, State: Initialized, Role: FOLLOWER
I20260925 19:13:53.000272 18643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.000671 18643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:53.000633 18649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.000733 18649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.001855 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:52.994303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.27.174.125:39009)
0925 19:13:52.994341 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:52.994343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:52.994352 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:52.994381 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:52.994382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:52.994394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:52.994447 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:52.994943 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.995630 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.996312 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:52.996652 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:52.996724 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:52.998864 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:52.998866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:52.998866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:52.998875 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.000554 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.000948 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.000950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.000950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.001012 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.001301 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.001302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.001303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.001433 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.001626 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.001666 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.001753 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:13:53.003042 18660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:53.003237 18660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:53.003280 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:13:53.004421 18467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bae91d5714f64ac294d4dcfff21bfb83"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51942
I20260925 19:13:53.005694 18493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } attempt: 2
I20260925 19:13:53.002909 18575 master_runner.cc:186] Error getting master registration for 127.27.174.126:41877: OK, instance_id { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" instance_seqno: 1790363632663759 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:13:53.006042 18575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 19:13:53 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:13:53.013253 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.005971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53298)
0925 19:13:53.006009 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:53.006021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:53.006063 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:53.006225 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:53.006227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:53.006382 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:53.006428 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:53.006430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.007070 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.007071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:53.007561 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:53.007562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.007999 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.008000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:53.008057 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.008526 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:53.008532 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:53.009964 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:53.012103 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:53.012104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:53.012105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:53.012334 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:53.012540 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:53.012541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:53.012541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:53.012616 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:53.012928 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:53.012929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:53.012979 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 19:13:53.013054 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:13:53.013153 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 19:13:53.013263 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.006032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.27.174.124:38295)
0925 19:13:53.006071 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.006073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.006083 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:13:53.006131 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.006132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.006142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.006204 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.006504 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.006990 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.007710 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.007915 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.007999 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.010084 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:53.010086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:53.010086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:53.010094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.012018 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.012413 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.012414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.012414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.012472 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.012682 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.012682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.012683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.012854 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.012919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.013089 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.013165 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:13:53.015188 18493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: Bootstrap starting.
I20260925 19:13:53.015935 18493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:53.016224 18493 log.cc:824] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:53.016994 18493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: No bootstrap required, opened a new log
I20260925 19:13:53.018342 18493 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.018479 18493 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae91d5714f64ac294d4dcfff21bfb83, State: Initialized, Role: FOLLOWER
I20260925 19:13:53.018563 18493 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.018875 18493 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:53.018887 18467 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:13:53.018940 18467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:53.019800 18467 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.020054 18467 leader_election.cc:288] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [CANDIDATE]: Term 1 election: Requested vote from peers 0e7272354b9d41338aa22cffd25ac9d7 (127.27.174.125:39009), 21a47590e8d74ed0acbc113f62e638aa (127.27.174.124:38295)
I20260925 19:13:53.020723 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bae91d5714f64ac294d4dcfff21bfb83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21a47590e8d74ed0acbc113f62e638aa"
W20260925 19:13:53.020797 18674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:53.020838 18674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:53.020854 18617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:53.021531 18617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae91d5714f64ac294d4dcfff21bfb83 in term 1.
I20260925 19:13:53.021754 18444 leader_election.cc:302] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21a47590e8d74ed0acbc113f62e638aa, bae91d5714f64ac294d4dcfff21bfb83; no voters:
I20260925 19:13:53.021955 18676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:13:53.022079 18676 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [term 1 LEADER]: Becoming Leader. State: Replica: bae91d5714f64ac294d4dcfff21bfb83, State: Running, Role: LEADER
I20260925 19:13:53.022161 18676 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.022570 18676 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae91d5714f64ac294d4dcfff21bfb83. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.022643 18676 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:53.024487 18425 master_runner.cc:186] Error getting master registration for 127.27.174.125:39009: OK, instance_id { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" instance_seqno: 1790363632814668 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:13:53.026783 18425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:13:53.018838 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.027082 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.027386 18679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:13:53.027948 18679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:13:53.030437 18617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [term 1 FOLLOWER]: Refusing update from remote peer bae91d5714f64ac294d4dcfff21bfb83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:53.030911 18663 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:53.032594 18649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae91d5714f64ac294d4dcfff21bfb83. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.032696 18649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.032989 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.023621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.27.174.125:39009)
0925 19:13:53.023965 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.023967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.024256 (+ 289us) server_negotiation.cc:407] Connection header received
0925 19:13:53.024349 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.024349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.024360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.024392 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.025176 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.025902 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.026914 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.027223 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.027272 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.029352 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:53.029353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:53.029353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:53.029361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.031353 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.031945 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.031946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.031947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.032013 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.032291 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.032292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.032292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.032429 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.032510 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.032807 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.032890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 19:13:53.035840 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.035946 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: This master's current role is: LEADER
I20260925 19:13:53.036253 18649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.036331 18649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.036338 18679 catalog_manager.cc:1347] Generated new cluster ID: e2b23a64ec7a46f9a9829b09eafec03e
I20260925 19:13:53.036381 18679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:13:53.037540 18687 mvcc.cc:205] Tried to move back new op lower bound from 7333329440888270848 to 7333329440859529216. Current Snapshot: MvccSnapshot[applied={T|T < 7333329440888270848 or (T in {7333329440888270848})}]
I20260925 19:13:53.040659 18688 mvcc.cc:205] Tried to move back new op lower bound from 7333329440888270848 to 7333329440859529216. Current Snapshot: MvccSnapshot[applied={T|T < 7333329440888270848}]
I20260925 19:13:53.040844 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.024000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.27.174.124:38295)
0925 19:13:53.024033 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.024035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.027696 (+ 3661us) server_negotiation.cc:407] Connection header received
0925 19:13:53.027805 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.027806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.027816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.027846 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.032798 (+ 4952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.033534 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.034642 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.035026 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.035090 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.037673 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:53.037674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:53.037674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:53.037682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.039294 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.039821 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.039822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.039823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.039891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.040121 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.040121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.040122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.040254 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.040338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.040681 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.040761 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:13:53.047245 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.023431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.27.174.126:41877)
0925 19:13:53.023830 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.023833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.027431 (+ 3598us) server_negotiation.cc:407] Connection header received
0925 19:13:53.027916 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.027917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.027927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.027958 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.031462 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.032124 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.036143 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.036433 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.036500 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.043862 (+ 7362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:53.043864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:53.043864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:53.043872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.045611 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.046179 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.046180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.046180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.046236 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.046515 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.046515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.046516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.046668 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.046751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.047056 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.047147 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260925 19:13:53.050334 18568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } attempt: 3
I20260925 19:13:53.051812 18679 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:13:53.053185 18679 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 19:13:53 dist-test-slave-fcdn krb5kdc[18412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20260925 19:13:53.060233 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.050606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:38295 (local address 127.0.0.1:53328)
0925 19:13:53.050658 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:13:53.050668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:13:53.050706 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:13:53.051863 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:13:53.051865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:13:53.052048 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:13:53.052109 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:53.052111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.052955 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.052956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:53.053702 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:13:53.053705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.054166 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.054167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:13:53.054222 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.054747 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:13:53.054752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:13:53.056392 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:13:53.059051 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:53.059052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:53.059052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:53.059274 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:53.059513 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:13:53.059514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:13:53.059514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:13:53.059582 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:13:53.059923 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:13:53.059924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:13:53.059980 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 19:13:53.060033 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 19:13:53.060132 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:13:53.060573 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:53.050841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.27.174.124:38295)
0925 19:13:53.050903 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.050905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.050913 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:53.050935 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.050936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.050984 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.051059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.052184 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.052881 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.053786 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:53.054073 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:53.054153 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:53.056484 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:13:53.056485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:13:53.056486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:13:53.056492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:13:53.058942 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.059366 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.059367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.059368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.059435 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:13:53.059685 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:13:53.059686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:13:53.059686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:13:53.059822 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:13:53.059879 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:53.060404 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 19:13:53.060481 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:13:53.062057 18568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: Bootstrap starting.
I20260925 19:13:53.062840 18568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: Neither blocks nor log segments found. Creating new log.
I20260925 19:13:53.063181 18568 log.cc:824] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: Log is configured to *not* fsync() on all Append() calls
I20260925 19:13:53.063478 18679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83: Generated new TSK 0
I20260925 19:13:53.063730 18679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:13:53.072379 18568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: No bootstrap required, opened a new log
I20260925 19:13:53.074548 18568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.074709 18568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7272354b9d41338aa22cffd25ac9d7, State: Initialized, Role: FOLLOWER
I20260925 19:13:53.074792 18568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } }
I20260925 19:13:53.075114 18568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:13:53.076555 18541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:13:53.077464 18541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [term 1 FOLLOWER]: Refusing update from remote peer bae91d5714f64ac294d4dcfff21bfb83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:13:53.077780 18677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:13:53.076413 18542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bae91d5714f64ac294d4dcfff21bfb83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e7272354b9d41338aa22cffd25ac9d7"
I20260925 19:13:53.079841 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.079929 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.079977 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae91d5714f64ac294d4dcfff21bfb83. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.080022 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:13:53.080053 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.080097 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:13:53.084208 18705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:13:53.084268 18705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:13:53.085882 18542 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bae91d5714f64ac294d4dcfff21bfb83 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260925 19:13:53.088213 18496 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.086726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.27.174.126:41877)
0925 19:13:53.086764 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.086766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.087517 (+ 751us) server_negotiation.cc:407] Connection header received
0925 19:13:53.087679 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.087681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.087693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.087725 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.088141 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51956: BlockingRecv error: recv got EOF from 127.0.0.1:51956 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:13:53.089104 18647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.087033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.27.174.124:38295)
0925 19:13:53.087070 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.087072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.088569 (+ 1497us) server_negotiation.cc:407] Connection header received
0925 19:13:53.088680 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.088681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.088690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.088719 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.089006 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:13:53.089910 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.087356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.27.174.125:39009)
0925 19:13:53.087394 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.087396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.089439 (+ 2043us) server_negotiation.cc:407] Connection header received
0925 19:13:53.089534 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.089535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.089545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.089573 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.089860 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37906: BlockingRecv error: recv got EOF from 127.0.0.1:37906 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:13:53.090556 18708 mvcc.cc:205] Tried to move back new op lower bound from 7333329441022484480 to 7333329440859529216. Current Snapshot: MvccSnapshot[applied={T|T < 7333329440888270848}]
W20260925 19:13:53.091840 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.091190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.27.174.125:39009)
0925 19:13:53.091226 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.091228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.091330 (+ 102us) server_negotiation.cc:407] Connection header received
0925 19:13:53.091430 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.091431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.091447 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.091476 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.091801 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37920: BlockingRecv error: recv got EOF from 127.0.0.1:37920 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:13:53.094702 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.094003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.27.174.125:39009)
0925 19:13:53.094034 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.094036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.094247 (+ 211us) server_negotiation.cc:407] Connection header received
0925 19:13:53.094334 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.094335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.094344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.094372 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.094664 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37922: BlockingRecv error: recv got EOF from 127.0.0.1:37922 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 19:13:53.098532 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.097818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.27.174.125:39009)
0925 19:13:53.097848 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.097850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.098084 (+ 234us) server_negotiation.cc:407] Connection header received
0925 19:13:53.098174 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.098174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.098185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.098214 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.098493 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37936: BlockingRecv error: recv got EOF from 127.0.0.1:37936 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 19:13:53.108276 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.103698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.27.174.125:39009)
0925 19:13:53.103731 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.103732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.107780 (+ 4048us) server_negotiation.cc:407] Connection header received
0925 19:13:53.107879 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.107880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.107890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.107919 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.108231 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37946: BlockingRecv error: recv got EOF from 127.0.0.1:37946 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:13:53.118410 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.117532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.27.174.125:39009)
0925 19:13:53.117570 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.117572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.117852 (+ 280us) server_negotiation.cc:407] Connection header received
0925 19:13:53.117953 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.117954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.117966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.117996 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.118345 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37956: BlockingRecv error: recv got EOF from 127.0.0.1:37956 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:13:53.136849 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.135880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.27.174.125:39009)
0925 19:13:53.135925 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.135928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.136225 (+ 297us) server_negotiation.cc:407] Connection header received
0925 19:13:53.136333 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.136334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.136347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.136378 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.136789 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37964: BlockingRecv error: recv got EOF from 127.0.0.1:37964 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:13:53.171377 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.170367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.27.174.125:39009)
0925 19:13:53.170413 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.170416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.170743 (+ 327us) server_negotiation.cc:407] Connection header received
0925 19:13:53.170858 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.170859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.170872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.170904 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.171317 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37968: BlockingRecv error: recv got EOF from 127.0.0.1:37968 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:13:53.237586 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.236834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.27.174.125:39009)
0925 19:13:53.236896 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.236898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.237071 (+ 173us) server_negotiation.cc:407] Connection header received
0925 19:13:53.237104 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.237105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.237119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.237200 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.237518 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37970: BlockingRecv error: recv got EOF from 127.0.0.1:37970 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 19:13:53.367584 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.366911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.27.174.125:39009)
0925 19:13:53.367008 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.367011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.367024 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:13:53.367096 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.367097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.367107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.367178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.367534 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54290: BlockingRecv error: recv got EOF from 127.0.0.1:54290 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 19:13:53.577972 18767 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae91d5714f64ac294d4dcfff21bfb83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae91d5714f64ac294d4dcfff21bfb83" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41877 } } peers { permanent_uuid: "0e7272354b9d41338aa22cffd25ac9d7" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 39009 } } peers { permanent_uuid: "21a47590e8d74ed0acbc113f62e638aa" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 38295 } } }
I20260925 19:13:53.578145 18767 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae91d5714f64ac294d4dcfff21bfb83 [sys.catalog]: This master's current role is: LEADER
W20260925 19:13:53.625555 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:13:53.624883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.27.174.125:39009)
0925 19:13:53.624957 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:53.624959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:53.624968 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:13:53.624993 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:53.624993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:53.625004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:53.625056 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:13:53.625482 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54296: BlockingRecv error: recv got EOF from 127.0.0.1:54296 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:13:54.003749 18660 catalog_manager.cc:1259] Loaded cluster ID: e2b23a64ec7a46f9a9829b09eafec03e
I20260925 19:13:54.003846 18660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: loading cluster ID for follower catalog manager: success
I20260925 19:13:54.004644 18660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: acquiring CA information for follower catalog manager: success
I20260925 19:13:54.004968 18660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21a47590e8d74ed0acbc113f62e638aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:13:54.084713 18705 catalog_manager.cc:1259] Loaded cluster ID: e2b23a64ec7a46f9a9829b09eafec03e
I20260925 19:13:54.084791 18705 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: loading cluster ID for follower catalog manager: success
I20260925 19:13:54.085716 18705 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: acquiring CA information for follower catalog manager: success
I20260925 19:13:54.086057 18705 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e7272354b9d41338aa22cffd25ac9d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:13:54.141575 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:54.138752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.27.174.125:39009)
0925 19:13:54.138896 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:54.138898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:54.138961 (+ 63us) server_negotiation.cc:407] Connection header received
0925 19:13:54.139075 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:54.139077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:54.139152 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:54.139259 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:54.139415 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.139947 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.140754 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.140956 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.141030 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:54.141232 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:54.141338 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:54.141426 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:54.141449 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 19:13:54.141470 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260925 19:13:54.146375 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:54.143728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.27.174.124:38295)
0925 19:13:54.143828 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:54.143829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:54.143857 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:13:54.143930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:54.143931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:54.143964 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:54.144033 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:54.144151 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.144680 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.145600 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.145842 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.145912 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:54.146068 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:54.146180 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:54.146248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:54.146280 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 19:13:54.146299 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
I20260925 19:13:54.151186 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:54.148450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.27.174.124:38295)
0925 19:13:54.148491 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:54.148493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:54.148665 (+ 172us) server_negotiation.cc:407] Connection header received
0925 19:13:54.148743 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:54.148743 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:54.148752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:54.148803 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:54.148893 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.149608 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150362 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.150656 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150722 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:54.150889 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:54.150921 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:54.150968 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:54.151069 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:13:54.151090 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:13:54.151239 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:54.148255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.27.174.125:39009)
0925 19:13:54.148318 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:54.148321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:54.148484 (+ 163us) server_negotiation.cc:407] Connection header received
0925 19:13:54.148517 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:54.148518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:54.148529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:54.148591 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:54.148751 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.149362 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150419 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.150708 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150765 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:54.150948 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:54.150981 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:54.151036 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:54.151130 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:13:54.151158 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:13:54.151423 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:13:54.148156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.27.174.126:41877)
0925 19:13:54.148326 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 19:13:54.148328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:13:54.148336 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:13:54.148390 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:13:54.148391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:13:54.148440 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:13:54.148533 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:13:54.148658 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.149497 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150465 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:13:54.150730 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:13:54.150793 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:13:54.150980 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:13:54.151105 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:13:54.151309 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 19:13:54.151328 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:13:54.151351 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":53,"threads_started":1}
I20260925 19:13:54.153012 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18425
I20260925 19:13:54.158257 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18500
I20260925 19:13:54.163280 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwy_z0T/build/release/bin/kudu with pid 18575
2026-09-25T19:13:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2789 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5617 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267614 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST